Augmented reality device for adjusting focus region according to direction of user&#39;s view and operating method of the same

ABSTRACT

An augmented reality (AR) device including a variable focus lens of which a focal length may be changed by adjusting refractive power and adjusting the position of a focus adjustment region of the variable focus lens according to a direction of the user&#39;s view. The AR device may obtain an eye vector indicating a direction of the user&#39;s view using an eye tracker, adjust a refractive power of a first focus adjustment region of a first variable focus lens to change a focal length for displaying a virtual image, and complementarily adjust a refractive power of a second focus adjustment lens with respect to the adjusted refractive power of the first focus adjustment region.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on and claims the benefit of U.S. ProvisionalPatent Application No. 62/853,082, filed on May 27, 2019, in the UnitedStates Patent and Trademark Office, and claims priority under 35 U.S.C.§ 119 to Korean Patent Application No. 10-2019-0115473, filed on Sep.19, 2019, in the Korean Intellectual Property Office, the disclosures ofwhich are herein incorporated by reference in their entireties.

BACKGROUND 1. Field

The disclosure relates to augmented reality (AR) devices that provide avirtual image to be perceived as being displayed on a real world object,and more particularly, to AR devices that automatically adjust theposition and size of a focus region according to a direction of a user'sview and operating methods of the same.

2. Description of Related Art

Augmented Reality (AR) is a technology that overlays a virtual image ona physical environment space of the real world or a real world object,and is implemented by an AR device utilizing the AR technology (e.g.,smart glasses). The AR device may be utilized various circumstances forinformation retrieval, directions, camera photography, etc. even thoughsmart glasses are also worn as fashion items and may be mainly used foroutdoor activities.

A user of the AR device generally sees a scene through a see-throughdisplay disposed close to user's eyes while the AR device is being wornby the user. Here, the scene includes one or more real world objects inthe physical environment or space that the user sees directly throughthe eyes. The AR device may project a virtual image onto the see-throughdisplay or the user's eyes through the see-through display, and the usermay simultaneously view the real world object and the projected virtualimage through the see-through display.

When viewing the virtual image through the see-through display whilewearing the AR device, although a focal length controlled by the lens ofthe user's eye is adjusted to the see-through display where the virtualimage is displayed, a vergence distance of the user's eye that viewsleft and right eye images of the virtual image to which the cubic effectis provided according to a binocular disparity is formed farther orcloser to the see-through display, which causes an inconsistency betweenthe focal length and the vergence distance. Thus, a user who watches thevirtual image of the AR device for a long time or is sensitive may incurdizziness or motion sickness. This problem is calledvergence-accommodation conflict (VAC). A focus position on which thevirtual image is formed may be adjusted by placing an optical lenscapable of adjusting the refractive power in front of or behind thesee-through display and adjusting the vergence of the optical lens, andthus the above problem may be solved.

SUMMARY

Aspects of the embodiments relate to augmented reality (AR) devices thatinclude a variable focus lens capable of changing a focal length byadjusting refractive power and that adjust the position of a focusadjustment region of the variable focus lens according to a direction ofthe user's view and operating methods of the same.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments of the disclosure.

In accordance with an aspect of the disclosure, an augmented reality(AR) device includes a plurality of first variable focus lenses and aplurality of second variable focus lenses, wherein each of the firstvariable focus lenses and each of the second variable focus lenses is anelectrically tunable liquid crystal lens; a plurality of waveguides,wherein each of the waveguides is disposed between the plurality offirst variable focus lenses and the plurality of second variable focuslenses; a plurality of eye trackers configured to obtain a left eyevector by tracking a direction of the left eye of an user of theaugmented reality device and a right eye vector by tracking a directionof the right eye of the user of the augmented reality device; a displaymodule configured to project light of a virtual image toward thewaveguide; and one or more processors configured to determine a firstfocus adjustment region of a first left eye variable focus lens amongthe plurality of first variable focus lenses based on the left eyevector, determine a third focus adjustment region of a first right eyevariable focus lens among the plurality of first variable focus lensesbased on the right eye vector, obtain a gaze point based on the left eyevector and the right eye vector, adjust a refractive power of the firstfocus adjustment region and a refractive power of the third focusadjustment region based on the gaze point, and based on the adjustedrefractive power of the first focus adjustment region and the adjustedrefractive power of the third focus adjustment region, adjust arefractive power of a second focus adjustment region of a second lefteye variable focus lens among the plurality of second variable focuslenses and a refractive power of a fourth focus adjustment region of asecond right eye variable focus lens among the plurality of secondvariable focus lenses.

For example, the AR device may further comprises a memory storing aprogram comprising one or more instructions, the one or more processorsare further configured to perform operations by executing the one ormore instructions of the program stored in the memory.

For example, when a user wears the AR device, the first left eyevariable focus lens is disposed at a position spaced apart from theuser's left eye by a first distance, and the second left eye variablefocus lens is disposed at a position spaced apart from the user's lefteye by a second distance, the second distance is greater than the firstdistance.

For example, the one or more processors may be further configured tocontrol to apply a control voltage that generates a phase modulationprofile relating to a position corresponding to the first focusadjustment region to the first left eye variable focus lens, and, basedon the applied control voltage, adjust the refractive power of the firstfocus adjustment region by changing an angle at which liquid crystalmolecules arranged at a position of the first focus adjustment regionare arranged among liquid crystal molecules of the first left eyevariable focus lens.

For example, the AR device may further include a depth sensor configuredto measure a depth value of a real world object disposed at the gazepoint, wherein the processor may be further configured to obtain themeasured depth value of the real world object from the depth sensor andadjust the refractive power of the first focus adjustment region basedon the obtained depth value so as to adjust a focal length of thevirtual object which is a partial region of the virtual image.

For example, the one or more processors may be further configured toadjust the refractive power of the second focus adjustment region to bethe same as the refractive power of the first focus adjustment region ina direction opposite to a direction of the refractive power of the firstfocus adjustment region.

For example, the one or more processors may be further configured todetermine a position of the second focus adjustment region based on theleft eye vector such that the first focus adjustment region and thesecond focus adjustment region are aligned in a direction of the lefteye vector.

For example, a size of the first focus adjustment region may bedetermined based on a size of a virtual object that is a partial regionof the virtual image output through a left eye waveguide.

For example, a size of the second focus adjustment region may bedetermined based on a size of a virtual object that is a partial regionof the virtual image output through a left eye waveguide and a spaceddistance between the first left eye variable focus lens and the secondleft eye variable focus lens.

For example, a plurality of a first focus adjustment regions may beprovided on the first left eye variable focus lens, and wherein theprocessor may be further configured to adjust refractive powers of theplurality of first focus adjustment regions such that differentvergences are formed according to the plurality of first focusadjustment regions.

In accordance with another aspect of the disclosure, an AR deviceincludes a plurality of first variable a plurality of first variablefocus lenses and a plurality of second variable focus lenses, whereineach of the first variable focus lenses and each of the second variablefocus lenses is an electrically tunable liquid crystal lens; a pluralityof waveguides, wherein each of the waveguides is disposed between theplurality of first variable focus lenses and the plurality of secondvariable focus lenses; a plurality of eye trackers configured to obtaina left eye vector by tracking a direction of the left eye of an user ofthe augmented reality device and a right eye vector by tracking adirection of the right eye of the user of the augmented reality device;a display module configured to project light of a virtual image towardthe waveguide; and one or more processors configured to determine afirst focus adjustment region of a first left eye variable focus lensamong the plurality of first variable focus lenses based on the left eyevector obtained from the eye tracker and determine a third focusadjustment region of a first right eye variable focus lens among theplurality of first variable focus lenses based on the right eye vector,obtain a gaze point based on the left eye vector and the right eyevector, adjust a refractive power of the first focus adjustment regionand a refractive power of the third focus adjustment region based on thegaze point, to change a focal length of a real world object, adjust arefractive power of a second focus adjustment region of a second lefteye variable focus lens among the plurality of second variable focuslenses and refractive power of a fourth focus adjustment region of asecond right eye variable focus lens among the plurality of secondvariable focus lenses, and independently adjust the refractive power ofthe second focus adjustment region and the refractive power of thefourth focus adjustment region regardless of the refractive power of thefirst focus adjustment region and the refractive power of the thirdfocus adjustment region.

In accordance with another aspect of the disclosure, an operating methodof an AR device includes obtaining a left eye vector by tracking adirection of the left eye using a first eye tracker and obtaining aright eye vector by tracking a direction of the right eye using a secondeye tracker; determining a first focus adjustment region of a first lefteye variable focus lens based on the obtained left eye vector anddetermining a third focus adjustment region of a first right eyevariable focus lens based on the obtained right eye vector; obtaining agaze point based on the left eye vector and the right eye vector,adjusting a refractive power of the first focus adjustment region and arefractive power of the third focus adjustment region based on the gazepoint; based on the adjusted refractive power of the first focusadjustment region and the adjusted refractive power of the third focusadjustment region, adjusting a refractive power of a second focusadjustment region of a second left eye variable focus lens and arefractive power of a fourth focus adjustment region of a second righteye variable focus lens; and projecting light of a virtual image towarda waveguide using a display module.

For example, the adjusting of the refractive powers of the first focusadjustment region and the third focus adjustment region includes:applying a control voltage that generates a phase modulation profilerelating to a position corresponding to the first focus adjustmentregion to the first left eye variable focus lens; and based on theapplied control voltage, adjusting the refractive power of the firstfocus adjustment region by changing an angle at which liquid crystalmolecules arranged at a position of the first focus adjustment regionare arranged among liquid crystal molecules of the first left eyevariable focus.

For example, the adjusting of the refractive power of the first focusadjustment region and the refractive power of the third focus adjustmentregion includes: measuring a depth value of a real world object disposedon the gaze point using a depth sensor; and adjusting the refractivepower of the first focus adjustment region based on the measured depthvalue so as to adjust a focal length of the virtual object which is apartial region of the virtual image.

For example, the adjusting of the refractive power of the second focusadjustment region and the refractive power of the fourth focusadjustment region includes: adjusting the refractive power of the secondfocus adjustment region such that the second focus adjustment regionforms a vergence in a direction opposite to a direction of an adjustedvergence of the first focus adjustment region and adjusting therefractive power of the fourth focus adjustment region such that thefourth focus adjustment region forms a vergence in a direction oppositeto a direction of an adjusted vergence of the third focus adjustmentregion.

For example, the adjusting of the refractive power of the second focusadjustment region and the refractive power of the fourth focusadjustment region includes: adjusting the refractive power of the secondfocus adjustment region to be the same as adjusted refractive power ofthe first focus adjustment region in a direction opposite to a directionof the adjusted refractive power of the first focus adjustment region.

For example, the operating method may further include: determining thesecond focus adjustment region based on the left eye vector such thatthe first focus adjustment region and the second focus adjustment regionare aligned in a direction of the left eye vector.

For example, a size of the first focus adjustment region may bedetermined based on a size of a virtual object that is a partial regionof the virtual image output through the waveguide.

For example, a size of the second focus adjustment region may bedetermined based on a size of a virtual object that is a partial regionof the virtual image output through the waveguide and a spaced distancebetween the first left eye variable focus lens and the second left eyevariable focus lens.

For example, a plurality of a first focus adjustment regions may beprovided on the first left eye variable focus lens, and the adjusting ofthe refractive powers of the first focus adjustment region and the thirdfocus adjustment region includes: adjusting refractive powers of theplurality of first focus adjustment regions such that differentvergences are formed according to the plurality of first focusadjustment regions.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a diagram illustrating an augmented reality (AR) device thatdisplays a virtual image on a real world object, according to anembodiment of the disclosure;

FIG. 2 is a plan view of an AR device according to an embodiment of thedisclosure;

FIG. 3 is a block diagram illustrating elements of an AR deviceaccording to an embodiment of the disclosure;

FIG. 4 is a flowchart illustrating a method of operating an AR deviceaccording to an embodiment of the disclosure;

FIG. 5A illustrates an eye tracker of an AR device according to anembodiment of the disclosure;

FIG. 5B illustrates an eye tracker of an AR device according to anembodiment of the disclosure;

FIG. 5C is a diagram illustrating a three-dimensional (3D) eyeball modelwith respect to the direction of a user's view;

FIG. 5D is a reference diagram for describing a method of calibrating aneye tracker according to an embodiment of the disclosure;

FIG. 6A is a perspective view of a variable focus lens of an AR deviceaccording to an embodiment of the disclosure;

FIG. 6B is a perspective view for describing a method, performed by avariable focus lens of an AR device, of adjusting the refractive powerof a focus adjustment region according to an embodiment of thedisclosure;

FIG. 7A is a diagram for describing concept of vergence of a variablefocus lens of an AR device according to an embodiment of the disclosure;

FIG. 7B is a diagram for describing concept of vergence of a variablefocus lens of an AR device according to an embodiment of the disclosure;

FIG. 8 is a flowchart illustrating a method, performed by an AR device,of adjusting the refractive power of a variable focus lens to change afocal length according to an embodiment of the disclosure;

FIG. 9 is a perspective view illustrating a waveguide and a displaymodule of an AR device according to an embodiment of the disclosure;

FIG. 10A is a diagram illustrating a method, performed by an AR device,of changing a position of a focus adjustment region of a variable focuslens according to a direction of a user's view according to anembodiment of the disclosure;

FIG. 10B is a diagram illustrating a method, performed by an AR device,of changing a position of a focus adjustment region of a variable focuslens according to a direction of a user's view according to anembodiment of the disclosure;

FIG. 11A is a diagram illustrating a method, performed by an AR device,of changing positions of focus adjustment regions of variable focuslenses based on an eye vector according to an embodiment of thedisclosure;

FIG. 11B is a diagram illustrating a method, performed by an AR device,of changing positions of focus adjustment regions of variable focuslenses based on an eye vector according to an embodiment of thedisclosure;

FIG. 12A illustrates a method, performed by an AR device, of adjusting afocal length according to an embodiment of the disclosure;

FIG. 12B illustrates a virtual image displayed on a real world objectthrough the AR device according to an embodiment of the disclosure.

FIG. 13A is a diagram for describing a method, performed by an ARdevice, of adjusting a focal length according to an embodiment of thedisclosure;

FIG. 13B is a diagram of a virtual image displayed on a real worldobject through the AR device according to an embodiment of thedisclosure;

FIG. 14 is a flowchart of a method, performed by an AR device, ofadjusting refractive power of a focus adjustment region of a variablefocus lens based on a depth value of a real world object according to anembodiment of the disclosure;

FIG. 15A is a diagram illustrating a method, performed by an AR device,of determining a size of a focus adjustment region of a variable focuslens according to an embodiment of the disclosure;

FIG. 15B is a diagram illustrating a method, performed by an AR device,of determining a size of a focus adjustment region of a variable focuslens according to an embodiment of the disclosure;

FIG. 16 illustrates a variable focus lens that includes a plurality offocus adjustment regions according to an embodiment of the disclosure;and

FIG. 17 is a block diagram illustrating an AR device according to anembodiment of the disclosure.

DETAILED DESCRIPTION

Although the terms used in the disclosure have been described in generalterms that are currently used in consideration of the functions referredto in the disclosure, the terms are intended to encompass various otherterms depending on the intent of those skilled in the art, precedents,or the emergence of new technology. Also, some of the terms used hereinmay be selected by the applicant. In this case, these terms are definedin detail below. Accordingly, the terms used in the disclosure are notdefined based on the meaning of the term, not on the name of a simpleterm, but on the contents throughout the disclosure.

An expression used in the singular encompasses the expression in theplurality, unless a clearly different meaning is provided in thecontext. The terms including technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which the disclosure belongs.

Throughout the disclosure, the expression “at least one of a, b or c”indicates only a, only b, only c, both a and b, both a and c, both b andc, all of a, b, and c, or variations thereof.

Throughout the entirety of the specification of the disclosure, when acertain part includes a certain element, the term ‘including’ means thata corresponding element may further include other elements unless aspecific meaning opposed to the corresponding element is written. Theterm used in the embodiments of the disclosure such as “unit” or“module” indicates a unit for processing at least one function oroperation, and may be implemented in hardware, software, or in acombination of hardware and software.

According to the context, the expression “configured to” used herein maybe used as, for example, the expression “suitable for,” “having thecapacity to,” “designed to,” “adapted to,” “made to,” or “capable of.”The term “configured to” is not limited to only “specifically designedto” in hardware. Instead, the expression “a device configured to” maymean that the device is “capable of” operating together with anotherdevice or other elements. For example, a “processor configured to (orset to) perform A, B, and C” may mean a dedicated processor (e.g., anembedded processor) for performing a corresponding operation or ageneric-purpose processor (e.g., a central processing unit (CPU) or anapplication processor) which performs corresponding operations byexecuting one or more software programs which are stored in a memorydevice.

In the disclosure, a refractive index refers to a degree to which aluminous flux is reduced in a medium as compared to a vacuum.

In the disclosure, a refractive power refers to a force that redirects adirection of a ray of light or a light path by the curved surface of alens. The refractive power is the inverse of the focal length, and theunit of the refractive power is m−1 or diopter (D). The sign of therefractive power is positive (+) in case of a convex lens and negative(−) in case of a concave lens.

In the disclosure, a vergence is an index indicating a degree to whichlight converges or diverges. The vergence may be adjusted according tothe refractive power of the lens. In an embodiment of the disclosure, avariable focus lens may adjust the vergence by adjusting the refractivepower of the lens and changing the ray of light or the light path.

In the disclosure, a virtual image is a project image projected onto awaveguide by the emission surface of a display module of an augmentedreality (AR) device.

In the disclosure, a virtual object refers to a partial region of thevirtual image output through the waveguide. The virtual object mayrepresent information related to a real world object. The virtual objectmay include, for example, at least one of characters, numbers, symbols,icons, images, or animations.

Hereinafter, the disclosure will be described in detail by explainingembodiments of the disclosure with reference to the attached drawings.The disclosure may, however, be embodied in many different forms andshould not be construed as being limited to the embodiments of thedisclosure set forth herein.

Reference will now be made in detail to embodiments of the disclosure,examples of which are illustrated in the accompanying drawings.

FIG. 1 is a diagram illustrating an augmented reality (AR) device 100that displays a virtual object 20 on a real world object 10, accordingto an embodiment of the disclosure.

Referring to FIG. 1, a user may view the real world object 10 and thevirtual object 20 while wearing the AR device 100. The virtual object 20is a partial region of a virtual image generated by a display module 140(see FIGS. 2 and 3) of the AR device 100 and output through a waveguide130. The virtual object 20 may indicate information related to the realworld object 10. In the embodiment of the disclosure shown in FIG. 1,the virtual object 20 may be an image including a name, a model name,and price information about a toaster that is an example of the realworld object 10.

The user may view the real world object 10 and the virtual object 20through a lens 102. The lens 102 may include a first variable focus lens110, a second variable focus lens 120, and the waveguide 130.

When the user wears the AR device 100, the first variable focus lens 110may be disposed at a position closer to a user's eye 30, and the secondvariable focus lens 120 may be disposed farther and spaced apart fromthe user's eye 30 than the first variable focus lens 110. In anembodiment of the disclosure, the first variable focus lens 110 may bedisposed at a position spaced apart from the eye 30 by a first distance,and the second variable focus lens 120 may be disposed at a positionspaced apart from the eye 30 by a second distance greater than the firstdistance. Each of the first variable focus lens 110 and the secondvariable focus lens 120 may include a liquid crystal molecule and may beconfigured as an electrically tunable liquid crystal lens capable ofchanging focus according to an electrical driving signal.

The waveguide 130 may be disposed between the first variable focus lens110 and the second variable focus lens 120. The waveguide 130 is anoptical element including a transparent material that may be describedas a see-through display. The waveguide 130 may project light of avirtual image through the display module 140 (see FIG. 3). The waveguide130 may include a plurality of regions in which a diffraction grating isformed. The light of the virtual image projected toward the waveguide130 may be reflected in the waveguide 130 according to the principle oftotal reflection. In the light of the virtual image projected onto thewaveguide 130, a light path may be changed by the diffraction gratingformed in the plurality of regions such that the virtual object 20 isfinally output to the user's eye 30. The waveguide 130 may performfunctionality similar to a light guide plate that changes the light pathof the virtual image.

The first variable focus lens 110 and the second variable focus lens 120may respectively include first and second focus adjustment regions 112and 122 capable of locally adjusting focus by adjusting the arrangementangle of liquid crystal molecules disposed in a specific regionaccording to an applied control voltage. In an embodiment of thedisclosure, the first variable focus lens 110 may adjust the refractivepower of the first focus adjustment region 112 by changing thearrangement angle of the liquid crystal molecules disposed in the firstfocus adjustment region 112 according to the control voltage, therebyadjusting a vergence. The vergence is an index indicating a degree towhich light converges or diverges. The vergence may be adjustedaccording to the refractive power of a lens.

The first focus adjustment region 112 may adjust the vergence formed onthe first variable focus lens 110 by changing the refractive power likea concave lens, thereby changing a focal length by which the virtualobject 20 is formed. The first focus adjustment region 112 may changethe light path passing through the first focus adjustment region 112with the adjusted refractive power, and adjust the focal length of thevirtual object 20 projected onto the waveguide 130 to be the same as thefocal length of the real world object 10.

Although the focal length of the virtual object 20 changes by theadjusted refractive power of the first focus adjustment region 112 ofthe first variable focus lens 110, because the focal length with respectto the real world object 10 also changes, the real world object 10 isnot optimally focused, which may cause a problem in that the real worldobject 10 is perceived as dim by the user. To compensate for the focallength distortion of the real world object 10 caused by the vergencechange due to the adjusted refractive power of the first focusadjustment region 112 of the first variable focus lens 110, the ARdevice 100 may complementarily adjust the refractive power of the secondfocus adjustment region 122 of the second variable focus lens 120 withrespect to the refractive power of the first focus adjustment region112.

In an embodiment of the disclosure, the AR device 100 may adjust therefractive power of the second focus adjustment region 122 to be thesame as the adjusted refractive power of the first focus adjustmentregion 112 in a direction opposite to the direction of the refractivepower of the first focus adjustment region 112. For example, when thefirst focus adjustment region 112 is adjusted to the refractive power of−1 diopter D, the second focus adjustment region 122 may be adjusted to+1 diopter D. In an embodiment of the disclosure, the second variablefocus lens 120 may be an optical element that serves as a convex lens.

The AR device 100 may determine the positions of the first focusadjustment region 112 and the second focus adjustment region 122according to a direction of a user's view. In an embodiment of thedisclosure, the AR device 100 may obtain an eye vector indicating thedirection of the user's view by tracking the position of the user's eye30 using an eye tracker 150. The AR device 100 may determine thepositions of the first focus adjustment region 112 and the second focusadjustment region 122 according to according to the eye vector. In anembodiment of the disclosure, the AR device 100 may determine thepositions of the first focus adjustment region 112 and the second focusadjustment region 122 such that the first focus adjustment region 112and the second focus adjustment region 122 are aligned along thedirection of the eye vector.

The AR device 100 may determine the sizes of the first focus adjustmentregion 112 and the second focus adjustment region 122. In an embodimentof the disclosure, the AR device 100 may determine the sizes of thefirst focus adjustment region 112 and the second focus adjustment region122 based on the size of the virtual object 20. In another embodiment ofthe disclosure, the AR device 100 may determine the sizes of the firstfocus adjustment region 112 and the second focus adjustment region 122based on the size of the virtual object 20 and the distance between thefirst variable focus lens 110 and the second variable focus lens 120.

Conventionally, when viewing the virtual object 20 displayed through thewaveguide 130 while wearing the AR device 100, the focal length formedon the virtual object 20 is fixed to the distance between the waveguide130 and the eye 30, and the focal length is different from the focallength of the real world object 10 focused on a gaze point at which botheyes of the user converge according to a binocular disparity. This maycause a problem in that the virtual object 20 is displayed as dim orblurred and the user may experience dizziness or motion sickness. Toavoid an undesirable user experience, an optical method of adjusting thefocal length of the virtual object 20 by adjusting the refractive powerof an optical lens and changing the vergence may be utilized. However,when an optical lens with a fixed focus region is used, the focus maynot be adjusted according to the user's view, which causes a problem inthat the optical performance is poor.

In general, when the user looks at the center of the lens, the opticalperformance is excellent. The AR device 100 according to an embodimentof the disclosure may include the first variable focus lens 110 and thesecond variable focus lens 120, respectively including the first focusadjustment region 112 and the second focus adjustment region 122,capable of locally adjusting the vergence by changing refractive power.The AR device 100 may automatically adjust the positions of the firstfocus adjustment region 112 and the second focus adjustment region 122according to the eye vector obtained using the eye tracker 150, therebyadjusting the focal length of the virtual object 20 to be the same asthe focal length of the real world object 10. Accordingly, the AR device100 of the disclosure may display the virtual object 20 more clearly andimprove the optical performance of the AR device. In addition, the ARdevice 100 may automatically adjust the focal length of the virtualobject 20 according to the direction of the user's view, therebyalleviating the dizziness or motion sickness of the user and providingan improved user experience.

FIG. 2 is a plan view of the AR device 100 according to an embodiment ofthe disclosure.

Referring to FIG. 2, the AR device 100 may include a left eye lens 102,a right eye lens 104, a frame 106, a nose bridge 108, a display module140, a left eye tracker 150-1, a right eye tracker 150-2, a processor160, and a camera 170.

FIG. 2 is a diagram illustrating structural features of the AR device100. The structural features are described in FIG. 2, and operationsand/or functions of elements included in the AR device 100 are describedin detail in FIG. 3.

The left eye lens 102 may include a first left eye variable focus lens110-1, a second left eye variable focus lens 120-1, and a left eyewaveguide 130-1. The right eye lens 104 may include a first right eyevariable focus lens 110-2, a second right eye variable focus lens 120-2,and a right eye waveguide 130-2. Each of the first left eye variablefocus lens 110-1, the first right eye variable focus lens 110-2, thesecond left eye variable focus lens 120-1, and the second right eyevariable focus lens 120-2 may include a liquid crystal molecule and maybe configured as an electrically tunable liquid crystal lens capable ofchanging focus according to an electrical driving signal.

The first left eye variable focus lens 110-1 may be disposed in the lefteye lens 102 at a position closer to a user's left eye, and the secondleft eye variable focus lens 120-1 may be disposed in the left eye lens102 farther and spaced apart from the user's left eye than the firstleft eye variable focus lens 110-1. In an embodiment of the disclosure,when a user wears the AR device 100, the first left eye variable focuslens 110-1 may be spaced apart from the left eye by a first distance,and the second left eye variable focus lens 120-1 may be disposed to bespaced apart from the left eye by a second distance. The second distancemay be greater than the first distance.

The left eye waveguide 130-1 may be disposed between the first left eyevariable focus lens 110-1 and the second left eye variable focus lens120-1, and the right eye waveguide 130-2 may be disposed between thefirst right eye variable focus lens 110-2 and the second right eyevariable focus lens 120-2. The right eye waveguide 130-2 is the same asthe left eye waveguide 130-1 except for the disposition in the right eyevariable focus lens 110-2, and thus the left eye waveguide 130-1 will bedescribed below.

When the user wears the AR device 100, the left eye waveguide 130-1 mayinclude a transparent material in which a partial region of a rear sideis visible. The rear side of the left eye waveguide 130-1 refers to asurface that the user's eye faces when the user wears the AR device 100.Conversely, the front side of the left eye waveguide 130-1 refers to asurface (i.e., a side far from the user's eye) opposite to the rearside.

The left eye waveguide 130-1 may be configured as a flat plate of asingle layer or multilayer structure of the transparent material throughwhich light may be reflected therein and propagated. The left eyewaveguide 130-1 may include a plurality of regions that face theemission surface of the display module 140 to receive the light of aprojected virtual image from the display module 140, propagate thelight, change a light path, and finally output the light to the user'seye. A diffraction grating may be formed in the plurality of regions.The left eye waveguide 130-1 may perform functionality similar to alight guide plate. The shape and characteristics of the left eyewaveguide 130-1 will be described in detail with reference to FIG. 9.

The frame 106 may be a support structure of the AR device 100 mounted tothe user's head when the user wears the AR device 100. The frame 106 mayhave equipped thereon the display module 140, the processor 160, and thecamera 170. The frame 106 may include an electrical wire for electricalconnection between the display module 140, the processor 160, and thecamera 170.

The nose bridge 108 is a support connecting the left eye lens 102 andthe right eye lens 104 and may be supported by the user's nose when theuser wears the AR device 100. In an embodiment of the disclosure, theframe 106 or the nose bridge 108 may have a built-in microphone thatrecords sound and transmits a recorded voice signal to the processor160.

The display module 140 may project light of the virtual image toward theleft eye waveguide 130-1. The display module 140 may be disposed on theframe 106, but the disposition of the display module 140 is not limitedthereto. The display module 140 may couple the virtual image generatedby the processor 160 with the light to project the coupled the virtualimage onto the left eye waveguide 130-1 and the right eye waveguide130-2 through the emission surface. In an embodiment of the disclosure,the display module 140 may perform functionality similar to a projector.

The display module 140 may include a light source and an image panel.The light source is an optical element that illuminates the light andmay generate the light by adjusting colors of RGB. The light source maybe configured as, for example, a light emitting diode (LED). The imagepanel may be configured as a reflective image panel that modulates andreflects the light illuminated by the light source to light including atwo-dimensional (2D) image. The reflective image panel may be, forexample, a digital micromirror device (DMD) panel or a liquid crystal onsilicon (LCoS) panel, or another known reflective image panel.

The display module 140 may obtain image data constituting the virtualimage from the processor 160, generate the virtual image based on theobtained image data, couple the virtual image with the light output fromthe light source, and project lights of the coupled virtual image towardthe left eye waveguide 130-1 and the right eye waveguide 130-2 throughthe emission surface. In an embodiment of the disclosure, the processor160 may provide the image data including RGB color and luminance valuesof a plurality of pixels constituting the virtual image to the displaymodule 140. The display module 140 may perform image processing usingthe RGB color value and the luminance value of each of the plurality ofpixels and control the light source, thereby projecting light of thevirtual image onto the left eye waveguide 130-1 and the right eyewaveguide 130-2.

The display module 140 may be configured as a plurality of displaymodules to project the virtual image according to both eyes. In anembodiment of the disclosure, the AR device 100 may include the displaymodule 140 that projects the virtual image onto the right eye waveguide130-2. However, the display module 140 is not limited thereto and may beconfigured as one display module. When the AR device 100 includes onedisplay module 140, the AR device 100 may further include a reflectivemember that reflects light of the virtual image projected from thedisplay module 140 and propagates the reflected light of the virtualimage to the left eye waveguide 130-1 and the right eye waveguide 130-2.For example, the reflective member may be configured as a mirror.

The display module 140 will be described in detail with reference toFIG. 9.

The left eye tracker 150-1 may be disposed in a left portion of theframe 106 supporting the left eye lens 102. The right eye tracker 150-2may be disposed in a right portion of the frame 106 supporting the righteye lens 104. The left eye tracker 150-1 may obtain a first eye vectorindicating a direction of the left eye's view by tracking the positionand the direction of the user's left eye. The right eye tracker 150-2may obtain a second eye vector indicating a direction of the right eye'sview by tracking the position and the direction of the user's right eye.

In an embodiment of the disclosure, the left eye tracker 150-1 and theright eye tracker 150-2 may obtain the eye vector of the user accordingto a technique of detecting a direction of view using the cornealreflection of infrared rays. In an embodiment of the disclosure, theleft eye tracker 150-1 may include an infrared irradiator 152 and aninfrared detector 154, and the right eye tracker 150-2 may include aninfrared irradiator 156 and an infrared detector 158. The infraredirradiators 152 and 156 may irradiate the infrared light to the cornealportions of the left and right eyes, respectively, and the infrareddetectors 154 and 158 may detect the infrared light reflected from thecorneas of the left and right eyes. The left eye tracker 150-1 and theright eye tracker 150-2 may determine directions of user's both eyesview through an amount of infrared light detected by the infrareddetectors 154 and 158, respectively, and obtain the eye vectorsindicating the respective directions of view of each of the eyes. Theleft eye tracker 150-1 may provide the processor 160 with the first eyevector. The right eye tracker 150-2 may provide the processor 160 withthe second eye vector.

The processor 160 may be located within the frame 106. The processor 160may be composed of one or a plurality. The processor 160 may obtain thefirst eye vector and the second eye vector from the left eye tracker150-1 and the right eye tracker 150-2, and estimate the gaze point Gviewed through both eyes based on the first eye vector and second eyevector. In an embodiment of the disclosure, the processor 160 maycalculate a three-dimensional (3D) position coordinate value of the gazepoint G based on the first eye vector and the second eye vector. In anembodiment of the disclosure, the processor 160 may determine a focusposition based on the 3D position coordinate value of the gaze point G.

The processor 160 may determine the positions of the first focusadjustment region 112 and the second focus adjustment region 122 suchthat the first focus adjustment region 112 and the second focusadjustment region 122 are aligned along the first eye vector. In anembodiment of the disclosure, the processor 160 may obtain atwo-dimensional (2D) position coordinate value of a region in which thefirst eye vector arrives in the entire region of the first left eyevariable focus lens 110-1, and determine the position of the first focusadjustment region 112 based on the 2D position coordinate value.Similarly, the processor 160 may obtain a 2D position coordinate valueof a region in which the first eye vector arrives in the entire regionof the second left eye variable focus lens 120-1, and determine theposition of the second focus adjustment region 122 based on the 2Dposition coordinate value.

The processor 160 may determine the positions of the third focusadjustment region 114 and the fourth focus adjustment region 124 suchthat the third focus adjustment region 114 and the fourth focusadjustment region 124 are aligned along the second eye vector. In anembodiment of the disclosure, the processor 160 may determine theposition of the first focus adjustment region 112 by changing the phaseof a control voltage applied to the first left eye variable focus lens110-1, and may determine the position of the second focus adjustmentregion 122 by changing the phase of a control voltage applied to thesecond left eye variable focus lens 120-1. The processor 160 maydetermine a region corresponding to the first focus adjustment region112 by adjusting the applied control voltage through a plurality ofexcitation electrodes disposed on a liquid crystal layer of the firstleft eye variable focus lens 110-1. A specific method performed by theprocessor 160 of determining the position of a focus adjustment regionaccording to the phase change of the control voltage will be describedin detail with reference to FIGS. 6A and 6B.

The processor 160 may change the focal length of the virtual objectbased on the 3D position information value of the gaze point G. In anembodiment of the disclosure, the processor 160 may calculate a distanceat which the eyes of both eyes converge according to the binoculardisparity, that is, the convergence distance d_(con) between the user'seyes 30 and the gaze point G. Thus, the processor 160 may change thefocal length d_(f) of the virtual object based on the convergencedistance d_(con). In an embodiment of the disclosure, the processor 160may change the vergence of the first left eye variable focus lens 110-1by adjusting the refractive power of the first focus adjustment region112 of the first left eye variable focus lens 110-1 and may change thevergence of the first right eye variable focus lens 110-2 by adjustingthe refractive power of the third focus adjustment region 114 of thefirst right eye variable focus lens 110-2.

In an embodiment of the disclosure, the processor 160 may adjust therefractive power of the first focus adjustment region 112 by changingthe arrangement angle of liquid crystal molecules disposed in the regioncorresponding to the position of the first focus adjustment region 112.The processor 160 may change the light path of the virtual objectdisplayed on the left eye waveguide 130-1 and transmitting an eye lens32 of the left eye by adjusting the refractive power of the first focusadjustment region 112. Because the light path of the virtual objectchanges, the focal length d_(f) of the virtual object formed on a retina34 of the left eye may change. Because the processor 160 adjusts therefractive power of the first focus adjustment region 112, the vergenceof the first left eye variable focus lens 110-1 may change, and thus thefocal length d_(f) physically formed on the left eye waveguide 130-1 maybe adjusted to be the same as the convergence distance d_(con). Theprocessor 160 may also adjust the refractive power of the third focusadjustment region 114 with respect to the first right eye variable focuslens 110-2 in the same manner as the first left eye variable focus lens110-1, and thus the virtual object formed on a retina 36 of the righteye may also be changed in the same manner.

Because the vergence of the first left eye variable focus lens 110-1changes by the adjustment of the refractive power of the first focusadjustment region 112, a focus distortion, in which a real world objectappears unfocused, may occur. To compensate for the focus distortion,the processor 160 may adjust the refractive power of the second focusadjustment region 122 of the second left eye variable focus lens 120-1.In an embodiment of the disclosure, the processor 160 may adjust therefractive power of the second focus adjustment region 122 such that thesecond focus adjustment region 122 forms a complementary vergence withrespect to the vergence due to the adjusted refractive power of thefirst focus adjustment region 112. In an embodiment of the disclosure,the processor 160 may adjust the refractive power of the second focusadjustment region 122 to be the same as the adjusted refractive power ofthe first focus adjustment region 112 in a direction opposite to thedirection of the refractive power of the first focus adjustment region112. For example, when the first focus adjustment region 112 is adjustedto the refractive power of −1 diopter D, the second focus adjustmentregion 122 may be adjusted to +1 diopter D.

The camera 170 may be disposed on the frame 106. The camera 170 mayobtain video data and still image data by capturing a physicalenvironment or a space. The camera 170 may transmit the obtained videodata and still image data to the processor 160. In an embodiment of thedisclosure, the camera 170 may store the video data and the still imagedata in a storage 190 (see FIG. 3).

FIG. 3 is a block diagram illustrating elements of the AR device 100according to an embodiment of the disclosure.

Referring to FIG. 3, the AR device 100 may include the first variablefocus lens 110, the second variable focus lens 120, the waveguide 130,the display module 140, an eye tracker 150, the processor. 160, a memory162, the camera 170, a depth sensor 172, a position sensor 180, and thestorage 190. The first variable focus lens 110, the second variablefocus lens 120, the display module 140, the eye tracker 150, theprocessor 160, the memory 162, the camera 170, the depth sensor 172, andthe storage 190 may be electrically and/or physically connected to eachother. Elements having the same reference numerals as the elements shownin FIG. 2 among the elements shown in FIG. 3 are the same as theelements shown in FIG. 2. Therefore, redundant descriptions will beomitted.

Each of the first variable focus lens 110 and the second variable focuslens 120 may include liquid crystal molecules and may be configured asan electrically tunable liquid crystal lens capable of changing focusaccording to an electrical driving signal.

Each of the first variable focus lens 110 and the second variable focuslens 120 may change an arrangement angle of the liquid crystal moleculesdisposed in a specific region according to an applied control voltage,and thus the position of a focus adjustment region capable of locallychanging a refractive power may move on a lens. The control voltage maybe controlled by the processor 160 and may be applied to each of thefirst variable focus lens 110 and the second variable focus lens 120 bya voltage control circuit. This will be described in detail in thedescription with reference to FIGS. 6A and 6B.

The first variable focus lens 110 may be configured as a plurality ofvariable focus lenses. For example, when a user wears the AR device 100,the plurality of first variable focus lenses 110 may include the firstleft eye variable focus lens 110-1 disposed in a region corresponding tothe left eye and the first right eye variable focus lens 110-2 disposedin a region corresponding to the right eye. Although the first variablefocus lens 110 is illustrated as a single block including the first lefteye variable focus lens 110-1 and the first right eye variable focuslens 110-2, it is understood that the first left eye variable focus lens110-1 is disposed in a region corresponding to the left eye and thefirst right eye variable focus lens 110-2 is disposed in a regioncorresponding to the right eye, as illustrated in FIG. 2.

The second variable focus lens 120 may be configured as a plurality ofvariable focus lenses. For example, when the user wears the AR device100, the plurality of second variable focus lenses 120 may include thesecond left eye variable focus lens 120-1 disposed in the regioncorresponding to the left eye and the second right eye variable focuslens 120-2 disposed in the region corresponding to the right eye.Although the second variable focus lens 120 is illustrated as a singleblock including the second left eye variable focus lens 120-1 and thesecond right eye variable focus lens 120-2, it is understood that thesecond left eye variable focus lens 120-1 is disposed in a regioncorresponding to the left eye and the second right eye variable focuslens 120-2 is disposed in a region corresponding to the right eye, asillustrated in FIG. 2.

The waveguide 130 is an optical element formed of a transparentmaterial. When the user wears the AR device 100, the waveguide 130 mayinclude the transparent material in which a partial region of a rearside is visible. The waveguide 130 may be configured as a flat plate ofa single layer or multilayer structure of the transparent materialthrough which light may be reflected therein and propagated. Thewaveguide 130 may include a plurality of regions that face the emissionsurface of the display module 140 to receive the light of a projectedvirtual image from the display module 140. The light of the virtualimage projected onto the waveguide 130 may propagate in the waveguide130 according to the principle of total reflection. The waveguide 130may include a plurality of regions for changing the path of the lightand finally outputting the light to the user's eyes. A diffractiongrating may be formed in the plurality of regions. The waveguide 130 mayperform functionality that is similar to a light guide plate. A specificshape and characteristics of the waveguide 130 will be described indetail with reference to FIG. 9.

The waveguide 130 may be configured as a plurality of waveguides. Thewaveguide 130 may include the left eye waveguide 130-1 and the right eyewaveguide 130-2. Although the waveguide 130 is illustrated as a singleblock including the first left eye waveguide 130-1 and the right eyewaveguide 130-2, it is understood that the left eye waveguide 130-1 isdisposed in a region corresponding to the left eye and the right eyewaveguide 130-2 is disposed in a region corresponding to the right eye,as illustrated in FIG. 2.

In an embodiment of the disclosure, the first left eye variable focuslens 110-1, the second left eye variable focus lens 120-1, and the lefteye waveguide 130-1 may constitute the left eye lens 102 (see FIG. 2).Similarly, in an embodiment of the disclosure, the first right eyevariable focus lens 110-2, the second right eye variable focus lens120-2, and the right eye waveguide 130-2 may constitute the right eyelens 104 (see FIG. 2).

The display module 140 may project the light of the virtual image towardthe waveguide 130. In an embodiment of the disclosure, the displaymodule 140 may project the light of the virtual image using atransmissive projection technology in which a light source is modulatedby an optically active material illuminated as a white light. Thedisplay module 140 may project the same virtual image onto the left eyewaveguide 130-1 and the right eye waveguide 130-2 or may projectdifferent virtual images onto the left eye waveguide 130-1 and the righteye waveguide 130-2. Here, the virtual image may be generated by theprocessor 160. The display module 140 will be described in detail withreference to FIG. 9.

The eye tracker 150 may obtain an eye vector indicating a direction ofthe user's view by tracking the position and the direction of the user'seyes. In an embodiment of the disclosure, the eye tracker 150 may obtainthe eye vector of the user by using a technique of detecting thedirection of view using corneal reflection of infrared rays. The eyetracker 150 may provide the obtained eye vector to the processor 160.

The eye tracker 150 may include the left eye tracker 150-1 that obtainsa first eye vector about the left eye and the right eye tracker 150-2that obtains a second eye vector about the right eye. Although the eyetracker 150 is illustrated as a single block including the left eyetracker 150-1 and the right eye tracker 150-2, it is understood that theleft eye tracker 150-1 is disposed in a region corresponding to the lefteye for tracking thereof and the right eye tracker 150-2 is disposed ina region corresponding to the right eye for tracking thereof, asillustrated in FIG. 2. A specific method performed by the eye tracker150 of obtaining the eye vector with respect to the direction of view ofboth eyes will be described in detail in the descriptions of FIGS. 5A to5D.

The processor 160 may control overall functions and/or operationsperformed by the AR device 100 by executing one or more instructions ofa program stored in and read from the memory 162. The processor 160 mayinclude hardware elements that perform arithmetic, logic andinput/output operations and signal processing.

The processor 160 may include at least one hardware of, for example, acentral processing unit, a microprocessor, a graphics processing unit,application specific integrated circuits (ASICs), digital signalprocessors (DSPs), digital signals (DSPDs), digital signal processingdevices (DSPDs), programmable logic devices (PLDs), or fieldprogrammable gate arrays (FPGAs), but is not limited thereto.

The memory 162 may store the program including one or more instructions.The memory 162 may include at least one type of hardware device of, forexample, a flash memory type hardware device, a random access memory(RAM), a static random access memory (SRAM), a read-only memory (ROM),an electrically erasable programmable read (EPEROM), programmableread-only memory (PROM), magnetic memory, magnetic disk, or opticaldisk. The processor 160 may be composed of one or a plurality.

The processor 160 may change the arrangement angle of liquid crystalmolecules disposed in a first focus adjustment region by applying acontrol voltage to the first left eye variable focus lens 110-1 andaccordingly adjust the refractive power of the first focus adjustmentregion, thereby adjusting the refractive index of light passing throughthe first focus adjustment region. The processor 160 may adjust thevergence of the first left eye variable focus lens 110-1 by adjustingthe refractive power of the first focus adjustment region. The vergenceis an index indicating a degree by which light converges or diverges,and may be adjusted according to the refractive power of the lens. In anembodiment of the disclosure, the processor 160 may adjust the vergenceof the first left eye variable focus lens 110-1 by adjusting therefractive power of the first focus adjustment region in a firstdirection, and adjust the focal length of a virtual object VO. Whenadjusting the vergence of the first focus adjustment region in adivergence direction, the path of the light passing through the firstfocus adjustment region may increase such that the focal length of thevirtual object VO formed on the retina of the left eye may increase. Inan embodiment of the disclosure, the processor 160 may adjust thevergence of the first focus adjustment region, and thus the focal lengthof the virtual object VO may be adjusted to be the same as theconvergence distance of both eyes.

The processor 160 may change the arrangement angle of liquid crystalmolecules disposed in a third focus adjustment region by applying acontrol voltage to the first right eye variable focus lens 110-2, andaccordingly adjusting the refractive power of the third focus adjustmentregion.

In an embodiment of the disclosure, the processor 160 may calculate a 3Dposition coordinate value of a gaze point at which the directions ofview of both eyes converge, by using both a first eye vector obtainedthrough the left eye tracker 150-1 and a second eye vector obtainedthrough the right eye tracker 150-2. Accordingly, based on a convergencedistance which is the distance between both eyes and the gaze point, theprocessor 160 may adjust the refractive power of the first focusadjustment region of the first left eye variable focus lens 110-1 andthe third focus adjustment region of the first right eye variable focuslens 110-2. The processor 160 may adjust the refractive power of thefirst focus adjustment region to change the focal length of the virtualobject VO displayed through the left eye waveguide 130-1 to be the sameas the convergence distance. Similarly, the processor 160 may adjust therefractive power of the third focus adjustment region to change thefocal length of the virtual object VO displayed through the right eyewaveguide 130-2 to be the same as the convergence distance.

In an embodiment of the disclosure, the processor 160 may storeinformation about the 3D position coordinate value of the gaze point inthe memory 160 and/or the storage 190.

In an embodiment of the disclosure, the first left eye variable focuslens 110-1 may include a plurality of first focus adjustment regions andthe processor 160 may adjust the refractive power of the plurality offirst focus adjustment regions to have focal lengths different from eachother according to the plurality of first focus adjustment regions. Inan embodiment of the disclosure, the first right eye variable focus lens110-2 may include a plurality of third focus adjustment regions, and theprocessor 160 may adjust the refractive power of the plurality of thirdfocus adjustment regions to have focal lengths different from each otheraccording to the plurality of third focus adjustment regions.

The processor 160 may change the arrangement angle of liquid crystalmolecules disposed in a second focus adjustment region by applying acontrol voltage to the second left eye variable focus lens 120-1 andaccordingly adjust the refractive index of light passing through thesecond focus adjustment region, thereby adjusting the refractive powerof the second left eye variable focus lens 120-1. In an embodiment ofthe disclosure, to compensate for a focus distortion in which a realworld object is perceived as blurry due to the adjusted refractive powerof the first focus adjustment region, the processor 160 controlling thesecond left eye variable focus lens 120-1 may complementarily adjust therefractive power of the second focus adjustment region of the secondleft eye variable focus lens 120-1 with respect to the adjustedrefractive power of the first focus adjustment region.

In an embodiment of the disclosure, the processor 160 may adjust therefractive power of the second focus adjustment region to be the same asthe adjusted refractive power of the first focus adjustment region in adirection opposite to the direction of the adjusted refractive power ofthe first focus adjustment region. For example, when the first focusadjustment region is adjusted to the refractive power of −1 diopter D,the processor 160 may adjust the second focus adjustment region to +1diopter D.

The processor 160 may adjust the refractive power of the fourth focusadjustment region to be the same as the adjusted refractive power of thethird focus adjustment region in a direction opposite to the directionof the refractive power of the third focus adjustment region such thatthe fourth focus adjustment region of the second right eye variablefocus lens 120-2 complementarily forms the vergence with respect to thevergence of the third focus adjustment region of the first right eyevariable focus lens 110-2.

In an embodiment of the disclosure, the processor 160 may independentlyadjust the refractive power of the second focus adjustment regionirrespective of the refractive power of the first focus adjustmentregion. In an embodiment of the disclosure, the processor 160 may adjustthe refractive power of the second focus adjustment region of the secondleft eye variable focus lens 120-1 to change the focus of the real worldobject viewed through the second left eye variable focus lens 120-1. Inan embodiment of the disclosure, when the user experiences hyperopia ormyopia, the processor 160 may adjust the refractive power of the secondfocus adjustment region of the second left eye variable focus lens 120-1and the fourth focus adjustment region of the second right eye variablefocus lens 120-2 for the purpose of correcting the user's vision. Inthis case, the processor 160 may adjust the refractive power of thesecond focus adjustment region and the fourth focus adjustment regionbased on the refractive index input by an external input or may adjustthe refractive power of the second focus adjustment region and thefourth focus adjustment region to have the refractive index previouslystored in the memory 162 and/or the storage 190.

The processor 160 may determine the position of the first focusadjustment region of the first variable focus lens 110 based on the eyevector obtained from the eye tracker 150. The processor 160 maydetermine the positions of the first focus adjustment region and thesecond focus adjustment region such that the first focus adjustmentregion of the first left eye variable focus lens 110-1 and the secondfocus adjustment region of the second left eye variable focus lens 120-1are aligned according to the direction of the first eye vector. In anembodiment of the disclosure, the processor 160 may obtain atwo-dimensional (2D) position coordinate value of a region in which thefirst eye vector arrives in the entire region of the first left eyevariable focus lens 110-1, and determine the position of the first focusadjustment region 112 based on the 2D position coordinate value.Similarly, the processor 160 may obtain a 2D position coordinate valueof a region in which the first eye vector arrives in the entire regionof the second left eye variable focus lens 120-1, and determine theposition of the second focus adjustment region 122 based on the 2Dposition coordinate value. Similarly, the processor 160 may determinethe positions of the third focus adjustment region and the fourth focusadjustment region such that the third focus adjustment region of thefirst right eye variable focus lens 110-2 and the fourth focusadjustment region of the first right eye variable focus lens 110-2 arealigned according to the direction of the second eye vector.

The processor 160 may determine sizes of the first focus adjustmentregion and the second focus adjustment region. In an embodiment of thedisclosure, the processor 160 may determine the sizes of the first focusadjustment region and the second focus adjustment region based on thesize of the virtual object VO projected onto the user's eye through theleft eye waveguide 130-1. In an embodiment of the disclosure, theprocessor 160 may determine the sizes of the first focus adjustmentregion and the second focus adjustment region based on the size of thevirtual object VO and the distance between the first left eye variablefocus lens 110-1 and the second left eye variable focus lens 120-1. Inan embodiment of the disclosure, the processor 160 may determine sizesof the third focus adjustment region and the fourth focus adjustmentregion based on the size of the virtual object VO projected through theright eye waveguide 130-2. In an embodiment of the disclosure, theprocessor 160 may determine the sizes of the third focus adjustmentregion and the fourth focus adjustment region based on the size of thevirtual object VO and the distance between the first right eye variablefocus lens 110-2 and the second right eye variable focus lens 120-2.

The camera 170 may obtain video data and still image data by capturing aphysical environment or a space viewed by the user. The camera 170 maytransmit the video data and still image data to the processor 160. In anembodiment of the disclosure, the camera 170 may visually monitor thesurrounding space of the user. In an embodiment of the disclosure, thecamera 170 may perform one or more controls or operations within anapplication controlled by the processor 160 or may capture a gesture ora motion performed by the user as well as the real world object of thesurrounding space to operate to provide input within the application.

The depth sensor 172 may be a sensor and/or camera that measures thedepth value of the real world object viewed by the user. The depthsensor 172 may scan the physical space or environment, measure the depthvalue of the real world object disposed in the physical space orenvironment according to a 3D position coordinate value of the realworld object, and measure the measured depth value in each 3D positioncoordinate value to generate a depth map. The depth sensor 172 may storethe depth map in the memory 162 and/or the storage 190.

In an embodiment of the disclosure, the depth sensor 172 may obtain adepth image that includes the depth value of the real world object. Thedepth image includes a 2D pixel region of a captured scene, in whicheach pixel in the 2D pixel region may represent, for example, a depthvalue such as the distance of the real world object in the capturedscene by using the depth sensor 172 in centimeters (cm), millimeters(mm), etc.

The depth sensor 172 may measure the 3D depth value by using any one of,for example, stereo-type, time-of-flight (ToF), and structured pattern.In an embodiment of the disclosure, the depth sensor 172 may include anRGB camera, an infrared light element, and a 3D camera that may be usedto capture the depth image about the real world object.

The depth sensor 172 may transmit the depth image including the depthvalue of the real world object to the processor 160 or store the depthimage in the memory 162 and/or the storage 190. In an embodiment of thedisclosure, the processor 160 may obtain the depth image from the depthsensor 172 or obtain the depth value of the real world object by loadingthe depth map stored in the memory 162 and/or the storage 190, andadjust the refractive power of the first focus adjustment region tochange the focal length of the virtual object VO based on the obtaineddepth value. In an embodiment of the disclosure, the processor 160 mayadjust the vergence of the first left eye variable focus lens 110-1 byadjusting the refractive power of the first focus adjustment region andadjust the focal length of the virtual object VO through the adjustmentof the vergence. A specific method of adjusting the focal length of thevirtual object VO using the depth value of the real world object will bedescribed in detail with reference to FIGS. 13A and 13B.

The position sensor 180 may obtain position information of the AR device100. In an embodiment of the disclosure, the position sensor 180 mayobtain a location or geographic position coordinates where the AR device100 is currently located. For example, the position sensor 180 mayinclude a GPS sensor.

The storage 190 may store at least one of the virtual image generated bythe processor 160, an image of the real world object captured by thecamera 170, or the depth image or the depth map with respect to the realworld object captured through the depth sensor 172. In an embodiment ofthe disclosure, the storage 190 may store information about therefractive power of the first focus adjustment region of the first lefteye variable focus lens 110-1 and the second focus adjustment region ofthe second left eye variable focus lens 120-1. In an embodiment of thedisclosure, the storage 190 may store information about the refractivepower of the third focus adjustment region of the first right eyevariable focus lens 110-2 and the fourth focus adjustment region of thesecond right eye variable focus lens 120-2.

The storage 190 may include, for example, at last one type storagemedium of a flash memory type memory, a hard disk type memory, amultimedia card micro type memory, a card type memory (e.g., SD or XDmemory, etc.), a magnetic memory, a magnetic disk, or an optical disk,but the storage 190 is not limited to the above-described example.

FIG. 4 is a flowchart illustrating a method of operating the AR device100 according to an embodiment of the disclosure.

In operation S410, the AR device 100 may obtain a first eye vector withrespect to a first direction of view of the left eye using a first eyetracker and obtain a second eye vector with respect to a seconddirection of view of the right eye using a second eye tracker. The firsteye tracker and the second eye tracker may obtain the first eye vectorand the second eye vector indicating the direction of the user's view bytracking the respective positions and directions of pupils of both eyesand provide the first eye vector and second eye vector to the AR device100. In an embodiment of the disclosure, an eye tracker may obtain theeye vector of the user by using a technique of detecting the directionof view using corneal reflection of infrared rays. In an embodiment ofthe disclosure, the eye tracker may obtain an image of the pupil using avision technology, track the position change of the pupil using theimage of the pupil, and obtain the eye vector based on the change inposition. The eye tracker may provide the processor 160 (see FIG. 3)with data about the vector value and the direction of each of theobtained first eye vector and second eye vector.

In operation S420, the AR device 100 may determine the position of afirst focus adjustment region of a first left eye variable focus lensbased on the first eye vector and may determine the position of a thirdfocus adjustment region of a first right eye variable focus lens basedon the second eye vector. The first left eye variable focus lens and thefirst right eye variable focus lens may be liquid crystal lenses capableof adjusting the refractive index of light passing through the firstfocus adjustment region and the third focus adjustment region bychanging the arrangement angle of liquid crystal molecules disposed inthe first focus adjustment region and the third focus adjustment region,respectively, according to control voltage phases applied thereto. In anembodiment of the disclosure, the first left eye variable focus lens andthe first right eye focus adjusting lens may be configured aselectrically tunable liquid crystal lenses capable of changing focusaccording to an electrical driving signal.

In an embodiment of the disclosure, the processor 160 (see FIG. 3) maycalculate position coordinate values of the first left eye variablefocus lens and the first right eye variable focus lens through which theeye of user's left and right eyes focus, based on the data about thevector value and the direction of each of the first eye vector andsecond eye vector obtained from the eye tracker. The processor 160 maydetermine each of the first focus adjustment region and the third focusadjustment region based on the calculated position coordinate value.

The AR device 100 may adjust the refractive power with respect to thefirst focus adjustment region, which is a region through which the firsteye vector passes in the entire region of the first left eye variablefocus lens. Similarly, the AR device 100 may adjust the refractive powerwith respect to the third focus adjustment region, which is a regionthrough which the second eye vector passes in the entire region of thefirst right eye variable focus lens. Because the direction of the firsteye vector changes, the position of the first focus adjustment regionmay change. Similarly, because the direction of the second eye vectorchanges, the position of the third focus adjustment region may change.

In operation S430, the AR device 100 may obtain a gaze point at whichthe first eye vector and the second eye vector converge according tobinocular disparity. In an embodiment of the disclosure, the gaze pointmay be obtained by using a triangulation method. The AR device 100 maycalculate a 3D position coordinate value of the gaze point based on thedistance between both eyes, the first eye vector, and the second eyevector. The AR device 100 may calculate a vergence distance, which is adistance between both eyes and the obtained gaze point.

In operation S440, the AR device 100 may adjust the refractive power ofthe first focus adjustment region and the third focus adjustment regionbased on the gaze point. In an embodiment of the disclosure, the ARdevice 100 may adjust the refractive power of the first focus adjustmentregion and the third focus adjustment region to adjust the focal lengthof a virtual object based on the vergence distance. Vergences formed inthe first focus adjustment region and the third focus adjustment regionmay be adjusted by adjusting the refractive power of the first focusadjustment region and the third focus adjustment region.

In an embodiment of the disclosure, the AR device 100 may obtain thevergence distance of a real world object disposed on the gaze point, andadjust the refractive power of the first focus adjustment region and thethird focus adjustment region based on the vergence distance such thatthe virtual object has the same focal length as the real world object.The AR device 100 may change the vergence formed in the first focusadjustment region to a divergence direction by adjusting the refractivepower of the first focus adjustment region, and accordingly adjust thefocal length of the virtual object formed on the retina of the left eye.Similarly, AR device 100 may change the vergence formed in the thirdfocus adjustment region to the divergence direction by adjusting therefractive power of the third focus adjustment region, and accordinglyadjust the focal length of the virtual object formed on the retina ofthe right eye.

In an embodiment of the disclosure, the processor 160 (see FIG. 3) mayadjust the refractive power of the first focus adjustment region bycontrolling an amount of control voltage applied to a plurality ofexcitation electrodes of the first left eye variable focus lens. Theprocessor 160 may apply the control voltage to an excitation electrodecorresponding to the position of the first focus adjustment region,which is a target region of which refractive power is to be adjustedamong the plurality of excitation electrodes of the first left eyevariable focus lens, and adjust the refractive power of the first focusadjustment region by adjusting the amount of the control voltage. Aspecific method performed by the processor 160 of determining theposition of a focus adjustment region of a focus adjustment lens andadjusting the refractive power of the focus adjustment region will bedescribed in detail with reference to FIGS. 6A and 6B.

In an embodiment of the disclosure, the AR device 100 may obtain a depthvalue of the real world object included in a depth image captured byusing a depth sensor and adjust the refractive power of the first focusadjustment region and the third focus adjustment region such that thefocal length of the virtual object is adjusted based on the obtaineddepth value of the real world object.

In operation S450, the AR device 100 may complementarily adjust therefractive power of each of the second focus adjustment region of thesecond left eye variable focus lens and the fourth focus adjustmentregion of the second right eye variable focus lens with respect to therefractive power of each of the first focus adjustment region and thethird focus adjustment region. In an embodiment of the disclosure, theAR device 100 may adjust the refractive power of the second focusadjustment region and the fourth focus adjustment region such that thesecond focus adjustment region and the fourth focus adjustment regioncomplementarily adjust the vergence with respect to the vergence of thefirst focus adjustment region and the third focus adjustment region,respectively. Although the focal length of the virtual object changes bythe vergence of the first focus adjustment region adjusted in operationS440, the focal length with respect to the real world object alsochanges by the vergence of the first focus adjustment region, which maycause a focus distortion in which the real world object looks blurry. Tocompensate for the focus distortion of the real world object due to theadjusted vergence of the first focus adjustment region of the first lefteye variable focus lens and the third focus adjustment region of thefirst right eye variable focus lens, the AR device 100 may adjust therefractive power of the second focus adjustment region of the secondleft eye variable focus lens and the fourth focus adjustment region ofthe second right eye variable focus lens to be the same as the adjustedrefractive power of the first focus adjustment region and the thirdfocus adjustment region in directions opposite to the directions of therefractive power of the first focus adjustment region and the thirdfocus adjustment region, respectively. For example, when the first focusadjustment region is adjusted to the refractive power of −1 diopter D,the AR device 100 may adjust the refractive power of the second focusadjustment region to +1 diopter D. Similarly, when the third focusadjustment region is adjusted to the refractive power of −2 diopter D,the AR device 100 may adjust the refractive power of the fourth focusadjustment region to +2 diopter D.

In operation S460, the AR device 100 may project light of a virtualimage toward a waveguide. The waveguide may be an optical element formedof a transparent material through which light may be reflected thereinand propagated. The waveguide may be configured as a flat plate of asingle layer or multilayer structure. The waveguide may perform afunction of a light guide plate that changes the direction of lightthrough the diffraction principle of the light and transmits the lightto the user's eye. The waveguide may include a first region in whichlight of the virtual image is received from the display module 140 (seeFIG. 3), a second region in which the light of the virtual imageincident on the first region propagates, and a third region that outputsthe virtual image by changing the direction of the light of the virtualimage propagated in the second region to the direction of the user'seye. A diffraction grating may be formed in the first region, the secondregion, and the third region. Due to the diffraction grating formed inthe first region, the second region, and the third region of thewaveguide, the light of the virtual image projected from the displaymodule 140 may be transmitted to the user's eye, and accordingly theuser may observe the virtual object of the virtual image. The structureof the waveguide and the light guiding principle will be described indetail with reference to FIG. 9.

The display module 140 (see FIG. 3) may project the light of the virtualimage toward the waveguide such that the virtual object is displayed atthe position coordinate value determined by the processor 160 (see FIG.3) based on the first eye vector and the second eye vector obtained bythe eye tracker.

The AR device 100 may determine the position of the virtual object whichis a partial region of the virtual image output through the waveguide130 (see FIG. 3) based on the eye vector. In an embodiment of thedisclosure, the processor 160 may obtain a 2D position coordinate valueof a region at which the first eye vector arrives in the entire regionof the left eye waveguide 130-1 from the eye tracker 150 and form thevirtual image such that the virtual object is displayed at the 2Dposition coordinate value. The processor 160 may provide generated imagedata of the virtual image to the display module 140. The processor 160may couple the virtual image with the light by controlling the lightsource and control the display module 140 to project the coupled virtualimage such that the virtual object of the virtual image is displayed ona region corresponding to the 2D position coordinate value of the lefteye waveguide 130-1.

FIG. 5A illustrates the eye tracker 150 of an AR device according to anembodiment of the disclosure.

Referring to FIG. 5A, the eye tracker 150 may include an infraredirradiator 152 and a plurality of infrared detectors 154 a, 154 b, 154c, 154 d, 154 e, and 154 f. Although six infrared detectors 154 a to 154f are shown in FIG. 5A, this is for convenience of description, and thequantity of the plurality of infrared detectors 154 a to 154 f and thepositioning thereof are not limited to this configuration.

The infrared irradiator 152 may irradiate an infrared light to thecornea portion corresponding to where the lens 32 of the eye 30 isdisposed. The plurality of infrared detectors 154 a to 154 f may detectthe infrared light reflected from the cornea. In an embodiment of thedisclosure, the infrared irradiator 152 may include a reflector thatchanges the path of the infrared light to irradiate the infrared lightto the direction of the eye 30. In an embodiment of the disclosure, theeye tracker 150 may obtain information about an amount of the infraredlight detected by each of the plurality of infrared detectors 154 a to154 f, determine a direction of view viewed by the user's eye 30 basedon the obtained amount of infrared light, and obtain an eye vectorindicating the direction of view. The eye tracker 150 may provide theprocessor 160 (see FIG. 3) with data indicating the vector value and thedirection of the obtained eye vector.

FIG. 5B illustrates an eye tracker of an AR device according to anembodiment of the disclosure.

Referring to FIG. 5B, the eye tracker 153 may track the eye of a userbased on positions of reflection light 511, 512, 513, 514, and 515reflected from the user's eye 30 and obtain an eye vector based on thepositions of reflection light 511, 512, 513, 514, and 515 reflected fromthe user's eye 30. The eye tracker 153 may include a light source 153 aand a camera 153 b.

The light source 153 a may include an infrared light emitting diode (IRLED). In the embodiment of the disclosure illustrated in FIG. 5B, thelight source 153 a may include a plurality of LEDs disposed at differentpositions. The light source 153 a may provide light (e.g., an infraredlight) to the user's eye 30 when an image of the eye 30 is captured bythe camera 153 b. Because the light is provided to the user's eye 30,the reflection light reflected from the user's eye 30 may be generated.

The camera 153 b may be configured to include at least one camera. Thecamera 153 b may be implemented as an infrared camera (IR). The ARdevice may track the view of the user's eye 30 using images 501, 502,503, 504, and 505 of the user's eye 30 captured by the camera 153 b. Forexample, the eye tracker 153 may track the user's view by detecting thepupil 500 and the reflection light 511 to 515 from the images 501 to 505of the user's eye 30, thereby obtaining an eye vector. The eye tracker153 may detect the positions of the pupil 500 and the reflection light511 to 515 from the images 501 to 505 of the user's eye 30 and determinethe direction of view of the user's eye 30 based on the relationshipbetween the position of the pupil 500 and the positions of thereflection light 511 to 515.

For example, the eye tracker 153 may detect the pupil 500 and thereflection light 511 from the captured first eye image 501 and determinea direction of view 521 of the user's eye 30 based on the relationshipbetween the position of the pupil 500 and the position of the reflectionlight 511. In the same manner, the eye tracker 153 may detect the pupil500 and the reflection light 512, 513, 514, and 515 respectively fromthe second to fifth eye images 502, 503, 504, and 505 and determinedirections of view 522, 523, 524, and 525 of the user's eye 30 based onthe relationships between the position of the pupil 500 and thepositions of the reflection lights 512, 513, 514, and 515.

In an embodiment of the disclosure, the eye tracker 153 may obtain theeye vector based on information about the direction of view. The eyetracker 153 may provide the processor 160 (see FIG. 3) with dataindicating the vector value and the direction of the eye vector.

In another embodiment of the disclosure, the eye tracker 153 may providethe processor 160 (see FIG. 3) with only coordinate values with respectto the position of the pupil 500 and the positions of the reflectionlight 511 to 515 detected from the plurality of eye images 501 to 505and the processor 160 may calculate the eye vector of the user's eye 30based on the coordinate values obtained from the eye tracker 153.

FIG. 5C is a diagram illustrating a 3D eyeball model with respect to thedirection of a user's view.

Referring to FIG. 5C, an AR device may determine the direction of viewof the user's eye 30 using the eye trackers 150 and 153. For example,the AR device may determine the direction of view based on an averageeyeball model of a person. The eyeball model may be modeled by assumingthat the human eye 30 has a spherical shape and the eyeball 30 ideallyrotates according to the direction of view. In addition, the eyeballmodel may be expressed mathematically as shown in Equations 1 and 2below.x=d·tan·α,y=d·sec α·tan β,  [Equation 1]β=sin⁻¹(diff_y/r),α=sin⁻¹(diff_x/r cos β).  [Equation 2]

In Equation 1, d denotes a distance between a center 33 of the user'seye 30 and a virtual screen 530, α denotes an angle of rotation of theuser's eye 30 in the x-axis direction based on an instance in which theuser's eye 30 gazes at the front of the virtual screen 530, and βdenotes an angle of rotation of the user's eye 30 in the y-axisdirection based on an instance in which the user's eye 30 gazes at thefront of the virtual screen 530. In addition, in Equation 2, r denotesthe radius of a sphere assuming that the user's eye 30 is the sphere.

The eye trackers 150 and 153 according to an embodiment of thedisclosure may use a method described in FIGS. 5A and 5B to measuredegrees of rotation (e.g., α and β) of the user's eye 30, and the ARdevice may use the degrees of rotation α and β of the user's eye 30 tocalculate a 2D coordinate value of the direction of view of the user'seye 30 on the virtual screen 530.

FIG. 5D is a reference diagram for describing a method of calibrating aneye tracker according to an embodiment of the disclosure.

Referring to FIG. 5D, when a user first uses an AR device, the AR devicemay perform a process of calibrating the eye trackers 150 and 153 toaccurately measure directions of view of the left and right eyes. The ARdevice may output virtual images VI1, VI2, and VI3 of different depths(e.g., d1, d2, and d3) marked with a plurality of points for guiding auser's view and induce the user to gaze at each of the plurality ofpoints. In FIG. 5D, although 9 points are shown in each of the virtualimages VI1, VI2, and VI3, this is an example and the quantity of pointsand positions of the points are not limited.

When the user gazes at each point included in the virtual images VI1,VI2, and VI3, the AR device may store data (e.g., an eye vector) outputfrom the eye trackers 150 and 153 in the storage 190 (see FIG. 3) in theform of a table.

As shown in FIG. 5A, the method of tracking a view using an amount ofinfrared light reflected from the cornea may previously storeinformation about the reflection angle and the amount of light at eachpoint in the storage 190 as view information in the form of the table.As shown in FIG. 5B, the method of capturing a user's eye using aninfrared light may previously store an image including the user's eyecaptured at each point and a reflection light in the storage 190 as viewinformation in the form of the table.

The AR device may determine the direction of view of the user's eye bycomparing previously stored view information with view informationoutput from the eye trackers 150 and 153. The processor 160 (see FIG. 3)of the AR device may determine directions of view of the left and righteyes using the view information output from the eye trackers 150 and153. In an embodiment of the disclosure, the processor 160 may calculatea first eye vector indicating a direction of view of the left eye and asecond eye vector indicating a direction of view of the right eye usingthe view information output from the eye trackers 150 and 153.

The AR device may estimate the position coordinate value of the gazepoint G (see FIG. 2) by using a binocular disparity and the viewinformation with respect to the direction of view of the left eye andthe direction of view of the right eye. For example, the processor 160(see FIG. 3) may use coordinate mapping to previously set a point (thegaze point G) at which the user gazes in the entire space described inFIG. 2 to be mapped to a 3D position coordinate value (e.g., an xcoordinate value, a y coordinate value, and a z coordinate value) or maystore the 3D position coordinate value of the gaze point G in thestorage 190 in the form of the table.

FIG. 6A is a perspective view of a variable focus lens of an AR deviceaccording to an embodiment of the disclosure.

Referring to FIG. 6A, the variable focus lens 600 may include a liquidcrystal layer 610, a common electrode 620, a transparent film 630, andfirst and second excitation electrodes 640 and 650. The variable focuslens 600 may further include a transparent layer formed in contact withthe lower surface of the common electrode 620.

The variable focus lens 600 may be an electrically tunable liquidcrystal lens capable of adjusting the refractive index of light bychanging an arrangement angle of liquid crystal molecules 612 based on acontrol voltage applied from a power supply VAC through the first andsecond excitation electrodes 640 and 650. In an embodiment of thedisclosure, the variable focus lens 600 may include an electro-opticmaterial having a pixel grid. A pixel may be arranged in a matrix of Nrows and M columns. Each N×M pixel may accommodate a set of possiblegray levels independent of all other pixels.

The liquid crystal layer 610 may be an electro-optical layer including aplurality of liquid crystal molecules 612. The liquid crystal layer 610may be an electro-optical layer of which a property of a liquid crystalchanges by the applied control voltage. In an embodiment of thedisclosure, the liquid crystal layer 610 may be configured as apolarization-independent liquid crystal layer (e.g., a cholestericliquid crystal). In the liquid crystal layer 610, the arrangement angleof the liquid crystal molecules 612 disposed in a specific region withinan active region changes by the control voltage applied through thefirst and second excitation electrodes 640 and 650, and thus therefractive index of the specific region may be locally adjusted.

The common electrode 620 and the first and second excitation electrodes640 and 650 may receive the control voltage from the power supply VACand apply the control voltage to the liquid crystal layer 610. Thecommon electrode 620 may be in contact with a first surface 610-1 of theliquid crystal layer 610.

The first and second excitation electrodes 640 and 650 may be disposedin contact with the upper surface of the transparent thin film 630 on asecond surface 610-2 opposing the first surface 610-1 of the liquidcrystal layer 610. Each of the first and second excitation electrodes640 and 650 may be respectively oriented in a direction orthogonal alongthe X-axis and Y-axis directions on the upper surface of the transparentthin film 630. Each of the first array excitation electrode 640 and thesecond array excitation electrode 650 may include the parallel strip ofa conductive material extending over the active region. In an embodimentof the disclosure, the first array excitation electrode 640 and thesecond array excitation electrode 650 may include a transparentconductive material such as indium tin oxide (ITO).

The pixel may be defined by a region at which the strip of the firstarray excitation electrode 640 and the strip of the second arrayexcitation electrode 650 overlap. The center-to-center distance betweenthe strip of the first array excitation electrode 640 and the strap ofthe second array excitation electrode 650 may define the pitch of apixel array, and the width of the strip defines may define the size ofthe pixel.

The processor 160 (see FIGS. 2 and 3) may control to apply a controlvoltage waveform having a phase modulation profile to each of the firstarray excitation electrode 640 and the second array excitation electrode650 through the power supply VAC and modulate the control voltageapplied to each of the first array excitation electrode 640 and thesecond array excitation electrode 650. The processor 160 maysimultaneously control to modulate the control voltage waveforms appliedto the first array excitation electrode 640 and the second arrayexcitation electrode 650 to generate a specific phase modulation profilein the liquid crystal layer 610.

Because the control voltage having a waveform modulated by the processor160 is applied, the refractive power of the variable focus lens 600 maybe locally adjusted in the specific region within the active region by aphase modulation profile of the applied control voltage. The variablefocus lens 600 may function as a lens in which vergence is adjustedaccording to the adjusted refractive power. Here, the vergence is anindex indicating a degree of convergence or divergence of light and maybe adjusted according to the refractive power of the variable focus lens600. In an embodiment of the disclosure, the variable focus lens 600 mayadjust the vergence by adjusting the refractive power of the lens tochange a ray of light or a light path.

The processor 160 may change a focal length by adjusting the vergence ofa specific region of the variable focus lens 600, that is, a focusadjustment region. A specific method of determining the position of aspecific region 612A (see FIG. 6B) of the variable focus lens 600 andadjusting the refractive power of the specific region 612A will bedescribed in detail with reference to FIG. 6B.

FIG. 6B is a perspective view for describing a method, performed by thevariable focus lens of an AR device, of adjusting the refractive powerof a focus adjustment region according to an embodiment of thedisclosure. In FIG. 6B, the variable focus lens 600 may include thefirst variable focus lenses 110-1 and 110-2 (see FIG. 2) or the secondvariable focus lenses 120-1 and 120-2 (see FIG. 2) of the left or righteye.

Referring to FIG. 6B, the variable focus lens 600 may include the liquidcrystal layer 610, the common electrode 620, a plurality of first arrayexcitation electrodes 640-1, 640-2, 640-3, 640-4, and 640-5, and aplurality of second array excitation electrodes 650-1, 650-2, 650-3,650-4, and 650-5. In FIG. 6B, unlike the case of FIG. 6A, thetransparent thin film 630 is not illustrated for convenience ofdescription.

The plurality of first array excitation electrodes 640-1 to 640-5 may bearranged along the X axis direction, and the plurality of second arrayexcitation electrodes 650-1 to 650-5 may be arranged along the Y axisdirection. The plurality of first array excitation electrodes 640-1 to640-5 and the plurality of second array excitation electrodes 650-1 to650-5 may be arranged to be orthogonal to each other.

A plurality of first driver terminals 660-1, 660-2, 660-3, 660-4, and660-5 that control a control voltage applied to the plurality of firstarray excitation electrodes 640-1 to 640-5 from the power supply VAC maybe connected to each of the plurality of first array excitationelectrodes 640-1 to 640-5. A plurality of second driver terminals 670-1,670-2, 670-3, 670-4, and 670-5 that control a control voltage applied tothe plurality of second array excitation electrodes 650-1 to 650-5 fromthe power supply VAC may be connected to each of the plurality of secondarray excitation electrodes 650-1 to 650-5.

A controller 680 for controlling operations of the variable focus lens600 may be connected to the plurality of first driver terminals 660-1 to660-5, the plurality of second driver terminals 670-1 to 670-5, and thepower supply VAC.

The controller 680 may control the control voltage applied to theplurality of first array excitation electrodes 640-1 to 640-5 and theplurality of second array excitation electrodes 650-1 to 650-5 bycontrolling the plurality of first driver terminals 660-1 to 660-5 andthe plurality of second driver terminals 670-1 to 670-5. Accordingly,the arrangement angle of liquid crystal molecules disposed in a specificregion may be controlled. in an embodiment, the variable focus lens 600may not include the plurality of first driver terminals 660-1 to 660-5and the plurality of second driver terminals 670-1 to 670-5 and thecontroller 680 may be directly connected to the plurality of first arrayexcitation electrodes 640-1 to 640-5 and the plurality of second arrayexcitation electrodes 650-1 to 650-5.

The controller 680 may receive view information including an eye vectorvalue indicating the direction of view of the user's eye and thedirection information of a vector from the processor 160 and determinethe position of the region A 612A of which focus is to be adjusted basedon the view information received from the processor 160. In anembodiment of the disclosure, the eye tracker 150 may obtain the eyevector by tracking the direction of view of the user's eye and providethe obtained eye vector to the processor 160. The processor 160 maycalculate a position coordinate value with respect to a region at whichthe view arrives in the entire region of the variable focus lens 600based on the vector direction of the eye vector and provide informationabout the position coordinate value to the controller 680. Thecontroller 680 may determine the region A 612A, which is a target regionof which focus is to be adjusted, based on the position coordinate valueobtained from the processor 160.

In the embodiment of the disclosure shown in FIG. 6B, to change thearrangement angle of liquid crystal molecules disposed in the region A612A among the plurality of liquid crystal molecules 612 included in theliquid crystal layer 610, the controller 680 may control voltages to beapplied to a 1-2th excitation electrode 640-2, a 1-3th excitationelectrode 640-3, and a 1-4th excitation electrode 640-4 among theplurality of first array excitation electrodes 640-1 to 640-5 and a2-2th excitation electrode 650-2, a 2-3th excitation electrode 650-3,and a 2-4th excitation electrode 650-4 among the plurality of secondarray excitation electrodes 650-1 to 650-5. In an embodiment of thedisclosure, the controller 680 may control the voltages to be applied tothe 1-2th excitation electrode 640-2, the 1-3th excitation electrode640-3, and the 1-4th excitation electrode 640-4 and the 2-2th excitationelectrode 650-2, the 2-3th excitation electrode 650-3, and the 2-4thexcitation electrode 650-4 by the power supply VAC by controlling a1-2th driver terminal 660-2 to a 1-4th driver terminal 660-4 and a 2-2thdriver terminal 670-2 to a 2-4th driver terminal 670-4. In thisconfiguration, the controller 680 may control voltages not to be appliedto a 1-1th excitation electrode 640-1 and a 1-5th excitation electrode640-5 by controlling a 1-1th driver terminal 660-1 and a 1-5th driverterminal 660-5 and may control voltages not to be applied to a 2-1thexcitation electrode 650-1 and a 2-5th excitation electrode 650-5 bycontrolling a 2-1th driver terminal 670-1 and a 2-5th driver terminal670-5.

The controller 680 may control whether to apply the control voltage fromthe power supply VAC and may control the magnitude of the controlvoltage applied from the power supply VAC. The controller 680 maycontrol the magnitude of the arrangement angle of the liquid crystalmolecules by controlling the magnitude of the applied control voltage.For example, when the controller 680 controls to apply the controlvoltage to the 1-2th excitation electrode 640-2 by a first magnitudethrough the control of the 1-2th driver terminal 660-2 and controls toapply the control voltage to the 1-3th excitation electrode 640-3 by asecond magnitude greater than the first magnitude through the control ofthe 1-3th driver terminal 660-3, the arrangement angle of the liquidcrystal molecules positioned in a region where the 1-3th excitationelectrode 640-3 is disposed in the entire region of the liquid crystallayer 610 may be adjusted to be greater than the arrangement angle ofthe liquid crystal molecules positioned in a region where the 1-2thexcitation electrode 640-2 is disposed.

That is, the controller 680 may determine a region in which thearrangement angle of the liquid crystal molecules 612 changes in theentire region of the liquid crystal layer 610 by modulating the phaseprofile of the control voltages applied to the plurality of first arrayexcitation electrodes 640-1 to 640-5 and the plurality of second arrayexcitation electrodes 650-1 to 650-5 through the plurality of firstdriver terminals 660-1 to 660-5 and the plurality of second driverterminals 670-1 to 670-5. Accordingly, the controller 680 may determinethe region A 612A of the variable focus lens 600 as a focus adjustmentregion. In addition, the controller 680 may adjust the refractive powerof the focus adjustment region of the variable focus lens 600 bymodulating the phase profile of the control voltage and adjusting thearrangement angle of the liquid crystal molecules 612 in the liquidcrystal layer 610.

In the above-described embodiment of the disclosure, a method ofadjusting the refractive power of the region corresponding to the regionA 612A of the variable focus lens 600 by changing the arrangement angleof the liquid crystal molecules 612 disposed in the region A 612A isdescribed. When the variable focus lens 600 is the first variable focuslens 110-1 or 110-2 (see FIG. 2) of the left eye or the right eye, aregion for adjusting the refractive power in the second variable focuslens 120-1 or 120-2 (see FIG. 2). That is, the focus control region, maybe determined differently according to the direction of view. Forexample, when the region A 612A is determined as the focus adjustmentregion in the first variable focus lens, a region aligned with theposition of the region A 612A of the first variable focus lens along thedirection of view in the entire region of the second variable focus lensmay be determined as the focus adjustment region. In an embodiment ofthe disclosure, information about the direction of view may be obtainedfrom the eye tracker 150. The processor 160 may use the informationabout the direction of view information obtained from the eye tracker150 to determine the position of the region aligned with the region A612A of the first variable focus lens along the direction of view in theentire region of the second variable focus lens.

In the embodiment of the disclosure illustrated in FIG. 6B, when thevariable focus lens 600 is the second variable focus lens, the regionaligned with the region A of the region 612A along the direction of viewmay be a region at which the 1-3th excitation electrode 640-3 to the1-5th excitation electrode 640-5 and the 2-1th excitation electrode650-1 to the 2-3th excitation electrode 650-3 intersect. To change thearrangement angle of the liquid crystal molecules arranged in the regionaligned with the region A of the region 612A along the direction of viewamong the plurality of liquid crystal molecules 612 included in theliquid crystal layer 610, the controller 680 may control voltages to beapplied to the 1-3th excitation electrode 640-3, the 1-4th excitationelectrode 640-4, and the 1-5th excitation electrode 640-5 among theplurality of first array excitation electrodes 640-1 to 640-5 and the2-1th excitation electrode 650-1, the 2-2th excitation electrode 650-2,and the 2-3th excitation electrode 650-3 among the plurality of secondarray excitation electrodes 650-1 to 650-5. In an embodiment of thedisclosure, the controller 680 may control the voltages to be applied tothe 1-3th excitation electrode 640-3, the 1-4th excitation electrode640-4, and the 1-5th excitation electrode 640-5 and the 2-1th excitationelectrode 650-1, the 2-2th excitation electrode 650-2, and the 2-3thexcitation electrode 650-3 by the power supply VAC by controlling the1-3th driver terminal 660-3 to the 1-5th driver terminal 660-5 and the2-1th driver terminal 670-1 to the 2-3th driver terminal 670-3, based onthe information about the direction of view received from the processor160. In this case, the controller 680 may control voltages not to beapplied to the 1-1th excitation electrode 640-1 and the 1-2th excitationelectrode 640-2 by controlling a 1-1th driver terminal 660-1 and a 1-2thdriver terminal 660-2 and may control voltages not to be applied to the2-4th excitation electrode 650-4 and the 2-5th excitation electrode650-5 by controlling a 2-4th driver terminal 670-4 and a 2-5th driverterminal 670-5.

By using the above-described method, the controller 680 may determinethe positions of the region A 612A of the first variable focus lens andthe focus adjustment region of the second variable focus lens and adjustthe refractive power of the determined focus adjustment region. Aspecific method of determining positions of the first focus adjustmentregion of the first variable focus lens and the second focus adjustmentregion of the second variable focus lens according to the direction ofview will be described in detail with reference to FIGS. 11A and 11B.

FIG. 7A is a conceptual diagram illustrating vergence formation of thevariable focus lens of an AR device according to an embodiment of thedisclosure.

Referring to FIGS. 7A and 7B, because a control voltage modulated tohave a specific phase profile is applied to the liquid crystal layer 610of the variable focus lens 600, the arrangement angle of the liquidcrystal molecules 612 disposed at a specific position in an activeregion may change. Because the arrangement angle of the liquid crystalmolecules 612 disposed at the specific region of the liquid crystallayer 610 changes, the refractive index of light passing through theliquid crystal molecules 612 may change. When the refractive index ofthe light changes, the refractive power of the variable focus lens 600changes, and the path of the light passing through the variable focuslens 600 changes, and thus a vergence may change. The vergence is anindex indicating a degree by which the light passing through thevariable focus lens 600 converges or diverges. The vergence may beadjusted according to the refractive power of the variable focus lens600.

In the embodiment of the disclosure shown in FIG. 7A, light passingthrough a region in which the alignment angle of the liquid crystalmolecules 612 included in the liquid crystal layer 610 changes may forma positive vergence, and thus the variable focus lens 600 may performfunctionality similar to the function of a convex lens. When thepositive vergence is formed, a focal length may be reduced.

FIG. 7B is a conceptual diagram illustrating vergence formation of thevariable focus lens of an AR device according to an embodiment of thedisclosure.

In the embodiment of the disclosure shown in FIG. 7B, light passingthrough a region in which the rotation angle of the liquid crystalmolecules 612 included in the liquid crystal layer 610 changes may forma negative vergence, and thus the variable focus lens 600 may performfunctionality similar to the function of a concave lens. When thenegative vergence is formed, the focal length may increase.

FIG. 8 is a flowchart illustrating a method performed by an AR device ofadjusting the refractive power of a variable focus lens and changing afocal length according to an embodiment of the disclosure.

In operation S810, the AR device may apply a control voltage thatgenerates a phase modulation profile relating to a positioncorresponding to a first focus adjustment region to a first left eyevariable focus lens. In an embodiment of the disclosure, the processor160 (see FIG. 6B) of the AR device may calculate a position coordinatevalue with respect to a region where the view of the user's eyes reachin the entire region of the variable focus lens, based on an eye vectorobtained by the eye tracker 150 (see FIG. 6B). The processor 160 mayprovide the calculated position coordinate value to the controller 680(see FIG. 6B) of the variable focus lens, and the controller 680 maydetermine the position of a first focus adjustment region that is atarget region of which focus is to be adjusted based on the positioncoordinate value.

The controller 680 may control to apply a control voltage waveformhaving a phase modulation profile to each of the plurality of firstarray excitation electrodes 640-1 to 640-5 (see FIG. 6B) and theplurality of second array excitation electrodes 650-1 to 650-5 (see FIG.6B) through the power supply VAC (see FIGS. 6A and 6B) and modulate thecontrol voltage applied to each of the plurality of first arrayexcitation electrodes 640-1 to 640-5 and the plurality of second arrayexcitation electrodes 650-1 to 650-5. The processor 160 may modulate thecontrol voltage through the controller 680 such that, among pixelsformed by overlapping the plurality of first array excitation electrodes640-1 to 640-5 and the plurality of second array excitation electrodes650-1 to 650-5, the pixels disposed in a region corresponding to thefirst focus adjustment region have phase values different from the otherpixels.

In operation S820, the AR device may change the angle at which liquidcrystal molecules are arranged at the position of the first focusadjustment region among the liquid crystal molecules of the first lefteye variable focus lens, based on the control voltage, thereby adjustingthe refractive power of the first focus adjustment region. The AR devicemay apply the control voltage having the phase modulation profile to thefirst left eye variable focus lens, thereby changing the arrangementangle of the liquid crystal molecules disposed in the regioncorresponding to the first focus adjustment region among the wholeliquid crystal molecules included in a liquid crystal layer. Because thearrangement angle of the liquid crystal molecules in the regioncorresponding to the first focus adjustment region changes, therefractive power of light passing through the first focus adjustmentregion may change. The AR device may adjust the refractive power of thefirst focus adjustment region by adjusting the arrangement angle of theliquid crystal molecules in the region corresponding to the first focusadjustment region.

In operation S830, the AR device may adjust the vergence of the firstfocus adjustment region through the refractive power. The AR device mayadjust the path of the light by adjusting the refractive power of thefirst focus adjustment region, thereby adjusting a degree of convergenceor divergence of the light. In an embodiment of the disclosure, the ARdevice may adjust the refractive power of the first focus adjustmentregion in a positive or negative direction, thereby correspondinglyreducing or increasing a focal length, which is the distance of an imageformed on the retina by passing through the lens of the eye. When therefractive power is adjusted such that the first focus adjustment regionhas a positive vergence, the first focus adjustment region may performfunctionality that is similar to a function of a convex lens. When therefractive power is adjusted such that the first focus adjustment regionhas a negative vergence, the first focus adjustment region may performfunctionality that is similar to a function of a concave lens.

In an embodiment of the disclosure, the refractive power of the firstfocus adjustment region may be adjusted in the negative direction toform the negative vergence. In this case, the focal length of a virtualobject output to the user's eye through the left eye waveguide 130-1 mayincrease.

The method of adjusting the vergence of the first focus adjustmentregion described with reference to FIG. 8 may also be applied to a firstright eye variable focus lens. The AR device of the disclosure mayadjust the refractive power of a third focus adjustment region of thefirst right eye variable focus lens by using the method illustrated inFIG. 8, thereby adjusting the vergence of the third focus adjustmentregion.

FIG. 9 is a perspective view illustrating the waveguide and the displaymodule of an AR device according to an embodiment of the disclosure.

Referring to FIG. 9, when a user wears the AR device, the waveguide 130may include a transparent material in which a partial region of a rearside is visible. The rear side of the waveguide 130 refers to a surfacethat the user's eye face when the user wears the AR device, and thefront side of the waveguide 130 refers to a surface (i.e., a side farfrom the user's eye) opposite to the rear side.

The waveguide 130 may be configured as a flat plate of a single layer ormultilayer structure of the transparent material through which light maybe reflected therein and propagated. The waveguide 130 may include afirst region 132 in which light of a virtual image VI projected byfacing an emission surface 142 of the display module 140 is received, asecond region 134 in which the light of the virtual image VI incident onthe first region 132 propagates, and a third region 136 that outputs thelight of the virtual image VI propagated in the second region 134 to thedirection of the user's eye. Here, the transparent material may be amaterial through which light may pass, and transparency thereof may notbe 100% and may have a predetermined color or tint.

In an embodiment of the disclosure, because the waveguide 130 includesthe transparent material, the user may view the virtual object of thevirtual image VI through the AR device and also view an external scene.Thus, the waveguide 130 may be referred to as a see through display. ARmay be implemented by outputting the virtual object of the virtual imageVI through the waveguide 130.

A diffraction grating may be formed in the first region 132, the secondregion 134, and the third region 136 to change the light path of thelight of the virtual image VI. The waveguide 130 may use the diffractiongrating formed in the first region 132, the second region 134, and thethird region 136 to change the propagation path of the light of thevirtual image VI, and may perform the function of a light guide platesuch that the light of the virtual image VI reflected through the thirdregion 136 may be output to the user's eyes.

A diffraction grating may be formed in the first region 132 to couplethe light of the virtual image VI incident from the emission surface 142of the display module 140 and transmit the light in the X-axisdirection. The display module 140 may be disposed such that the emittedlight is perpendicular to the first region 132 or is incident to beinclined at a predetermined angle. The arrangement direction of thedisplay module 140 may vary according to the pattern of the diffractiongrating of the first region 132.

The second region 134 may be spaced apart in the X-axis direction withrespect to the first region 132. A diffraction grating may be formed inthe second region 134 to propagate at least a part of the light receivedfrom the first region 132 downward along the Z-axis direction. When thewaveguide 130 is formed in a single layer structure, the diffractiongrating of the second region 134 may be formed on the same plane as thediffraction grating of the first region 132. Alternatively, when thewaveguide 130 is formed in a multilayer structure, the diffractiongrating of the second region 134 may be formed on a layer different fromthe layer on which the diffraction grating of the first region 132 isformed. The light incident on the first region 132 may propagate bybeing reflected between the front and rear sides of the waveguide 130.

The third region 136 may be spaced apart downward in the Z-axisdirection with respect to the second region 134. A diffraction gratingmay be formed in the third region 136 such that at least a part of thelight propagated from the second region 134 is output in atwo-dimensional (2D) plane. When the waveguide 130 is formed in a singlelayer structure, the diffraction grating of the third region 136 may beformed on the same plane as the diffraction gratings of the first region132 and the second region 134. Alternatively, when the waveguide 130 isformed in a multilayer structure, the diffraction grating of the thirdregion 136 may be formed on a layer different from the layer on whichthe diffraction grating of the second region 134 is formed and may beformed on the same layer as or a layer different from the diffractiongrating of the first region 132.

The diffraction grating of the first region 132 may have a differentpattern from the grating of the second region 134 and the diffractiongrating of the third region 136.

The display module 140 may couple the virtual image VI generated by theprocessor 160 (see FIGS. 2 and 3) with light to project the coupledvirtual image VI onto the waveguide 130 through the emission surface142. The display module 140 of the disclosure may perform the samefunction as a projector.

The display module 140 may further include an illumination opticalsystem, a light path converter, an image panel, a beam splitter, and aprojection optical system.

The illumination optical system is an optical element that illuminateslight and may include a light source and lenses. The light source is anelement that generates light by adjusting the color of RGB and may beconfigured as, for example, a light emitting diode (LED).

The image panel may be a reflective image panel that reflects andmodulates the light illuminated by the light source into light includinga 2D image. The reflective image panel may be, for example, a digitalmicromirror device (DMD) panel or a liquid crystal on silicon (LCoS)panel, or another known reflective image panel. The DMD panel mayoperate using a digital light processing (DLP) method of illuminatingthe RGB of the light output from the light source with a plurality ofmirrors each having a pixel size, switching each of the plurality ofmirrors on and off, and mixing the RGB of the light to project thevirtual image VI. The LCoS panel may operate using a liquid crystaldisplay (LCD) method of separating the light output from the lightsource into RGB through a mirror that passes only light of a specificwavelength, inputting the light to the image panel, and projecting thevirtual image VI generated by mixing the RGB.

The beam splitter may be disposed between the image panel and theprojection optical system. The beam splitter may be configured toreflect the light output from the light source and transmit the lightreflected by the image panel.

The projection optical system is an element that projects lightincluding the image reflected by the image panel onto the waveguide 130and may include a single or a plurality of projection lenses. In theembodiment of the disclosure shown in FIG. 9, the projection surface ofthe projection optical system refers to the emission surface 142 of theoutermost projection lens of a single or a plurality of projectionlenses.

The display module 140 may obtain image data constituting the virtualimage VI from the processor 160 (see FIGS. 2 and 3), generate thevirtual image VI based on the obtained image data, and couple thevirtual image VI with the light output from the light source to thecoupled virtual image VI onto the waveguide 130 through the emissionsurface 142. In an embodiment of the disclosure, the processor 160 mayprovide the display module 140 with the image data including RGB colorand luminance values of the plurality of pixels constituting the virtualimage VI. The display module 140 may project light of the virtual imageVI onto the waveguide 130 by performing image processing by using theRGB color value and the luminance value of each of the plurality ofpixels and controlling the light source.

In an embodiment of the disclosure, the display module 140 may generatethe virtual image VI using image data stored in the storage 190 (seeFIG. 3), couple the virtual image VI with light by controlling a lightsource, and project the light of the virtual image VI onto the waveguide130.

FIG. 10A is a diagram illustrating a method performed by an AR device ofchanging a position of a focus adjustment region of a variable focuslens according to a direction of the user's view according to anembodiment of the disclosure.

FIG. 10B is a diagram illustrating a method performed by an AR device ofchanging a position of a focus adjustment region of a variable focuslens according to a direction of the user's view according to anembodiment of the disclosure.

In FIGS. 10A and 10B, the first variable focus lens 110 may include thefirst focus adjustment region 112, and the second variable focus lens120 may include the second focus adjustment region 122. The waveguide130 may be disposed between the first variable focus lens 110 and thesecond variable focus lens 120 and project light or a virtual object VOof a virtual image. The virtual object VO may be a virtual objectdisplayed on a partial region of the virtual image.

Referring to FIG. 10A, the AR device may determine positions of thefirst focus adjustment region 112, the second focus adjustment region122, and the virtual object VO according to the direction of the user'sview. In an embodiment of the disclosure, the eye tracker 150 of the ARdevice may obtain an eye vector indicating the direction of view bytracking the position and direction of the user's eye 30. The eyetracker 150 may provide the eye vector to the processor 160 (see FIGS. 2and 3), and the processor 160 may determine the positions of the firstfocus adjustment region 112 and the second focus adjustment region 122such that the first focus adjustment region 112 and the second focusadjustment region 122 are aligned according to the direction of the eyevector. In an embodiment of the disclosure, the processor 160 may obtaina 2D position coordinate value of a region at which the eye vectorarrives in the entire region of the first variable focus lens 110, anddetermine the position of the first focus adjustment region 112 based onthe obtained 2D position coordinate value. Similarly, the processor 160may obtain the 2D position coordinate value of a region at which the eyevector arrives in the entire region of the second variable focus lens120, and determine the position of the second adjustment region 122based on the obtained 2D position coordinate value. The processor 160may determine a position at which the virtual object VO, which providesinformation related to a real world object in the virtual image, isdisplayed such that the virtual object VO is aligned with the firstfocus adjustment region 112 and the second focus adjustment region 122along the direction of the eye vector.

Referring to FIG. 10B, the AR device may change the positions of thefirst focus adjustment region 112 and the second focus adjustment region122 in response to changes of the direction of view of the user. Whenthe user's eye 30 looks in the horizontal direction and looks upward byα° with respect to the horizontal direction, the eye tracker 150 mayobtain the eye vector indicating the direction of view by tracking theposition of the eye 30 and the direction change and provide the eyevector to the processor 160. The processor 160 may change the positionsof the first focus adjustment region 112 and the second focus adjustmentregion 122 based on the eye vector. In an embodiment of the disclosure,the processor 160 may change the positions of the first focus adjustmentregion 112 and the second focus adjustment region 122 such that thefirst focus adjustment region 112 and the second focus adjustment region122 are aligned according to the direction of the eye vector.

In an embodiment of the disclosure, the AR device may modulate the phaseprofile of a control voltage by the processor 160 (see FIGS. 2 and 3)and apply the control voltage having the modulated phase profile to thefirst and second excitation electrodes 640 and 650 (see FIG. 6A),thereby changing the positions of the first focus adjustment region 112and the second focus adjustment region 122. The first variable focuslens 110 and the second variable focus lens 120 may include partialregions of the entire region where the focus is changed, that is, thefirst focus adjustment region 112 and the second focus adjustment region122. The positions of the first focus adjustment region 112 and thesecond focus adjustment region 122 change according to the direction ofthe eye vector, and thus the first variable focus lens 110 and thesecond variable focus lens 120 may perform functionality similar to afunction of a moving lens.

FIG. 11A is a diagram illustrating a method performed by an AR device ofchanging positions of the first focus adjustment region and the secondfocus adjustment region of the first variable focus lens and the secondvariable focus lens based on an eye vector according to an embodiment ofthe disclosure.

Referring to FIG. 11A, the AR device may obtain the eye vectorindicating a direction of the user's view as the user is looking at thereal world object 10. In an embodiment of the disclosure, an eye trackermay obtain a first eye vector p (u, v) representing the direction ofview. The eye tracker may provide the obtained eye vector p (u, v) tothe processor 160 (see FIGS. 2 and 3).

The AR device may obtain the gaze point G based on the binoculardisparity and the direction of the first eye vector p (u, v). Theprocessor 160 of the AR device may use the first eye vector p (u, v)obtained with respect to the binocular disparity and the left eye and asecond eye vector obtained with respect to the right eye to calculate a3D position coordinate value of the gaze point G. The processor 160 mayrecognize a virtual line/connecting the gaze point G from the user's eye30. The processor 160 may obtain coordinate information about a positionwhere the virtual line/meets the first variable focus lens 110 in theentire region of the first variable focus lens 110. The processor 160may determine a region including the position where the virtualline/meets on the first variable focus lens 110 as the first focusadjustment region 112. The first focus adjustment region 112 may beformed in a circular shape, but is not limited thereto.

The AR device may determine the position of the second focus adjustmentregion 122 based on the eye vector such that the first focus adjustmentregion 112 and the second focus adjustment region 122 are aligned in thedirection of the first eye vector p (u, v). In an embodiment of thedisclosure, the processor 160 may determine the position of the secondfocus adjustment region 122 such that the first focus adjustment region112 and the second focus adjustment region 122 are aligned along thevirtual line I connecting the first eye vector p (u, v) and the gazepoint G. In an embodiment of the disclosure, the first focus adjustmentregion 112, the second focus adjustment region 122, and the virtualobject VO may be aligned along the virtual line I.

The AR device may determine the size of the first focus adjustmentregion 112 based on the size of the virtual object VO of the virtualimage output through the waveguide 130.

In an embodiment of the disclosure, the AR device may determine the sizeof the second focus adjustment region 122 based on the spaced distancebetween the first variable focus lens 110 and the second variable focuslens 120. This will be described in detail with reference to FIGS. 15Aand 15B.

FIG. 11B is a diagram illustrating a method performed by an AR device ofchanging positions of the first focus adjustment region and the secondfocus adjustment region of the first variable focus lens and the secondvariable focus lens based on an eye vector according to an embodiment ofthe disclosure.

Referring to FIG. 11B, when the user looks at the real world object 10moving in the X-axis direction, the eye tracker may track a direction ofthe user's view and obtain a changed second eye vector p′ (u′, v′). Theeye tracker may provide the second eye vector p′ (u′, v′) to theprocessor 160. The processor 160 may change the positions of the firstfocus adjustment region 112 and the second focus adjustment region 122based on the changed second eye vector p′ (u′, v′).

The AR device may obtain the gaze point G using the second eye vector p′(u′, v′) and binocular disparity information. In an embodiment of thedisclosure, the processor 160 may recognize the virtual line/connectingthe user's eye 30 and the gaze point G and coordinate information aboutthe position where the virtual line/meets the first variable focus lens110 in the entire region of the first variable focus lens 110. Theprocessor 160 may determine a region including the position where thevirtual line/meets on the first variable focus lens 110 as the firstfocus adjustment region 112. The processor 160 may determine theposition of the second focus adjustment region 122 based on the eyevector such that the first focus adjustment region 112 and the secondfocus adjustment region 122 are aligned in the direction of the secondeye vector p′ (u′, v′). In an embodiment of the disclosure, theprocessor 160 may determine the positions of the second focus adjustmentregion 122 such that the first focus adjustment region 112 and thesecond focus adjustment region 122 are aligned along the virtual line Iobtained based on the second eye vector p′ (u′, v′).

Referring to FIGS. 11A and 11B, when the real world object 10 moves, theAR device may change the positions of the first focus adjustment region112 and the second focus adjustment region 122 based on the eye vectorobtained through the eye tracker. In another embodiment of thedisclosure, the direction of view may change when the user looks at theother real world object 10, in which case the AR device may track achange in the direction of the eye vector using the eye tracker andchange the positions of the first focus adjustment region 112 and thesecond focus adjustment region 122 according to the change in thedirection of the eye vector.

FIG. 12A illustrates a method of adjusting a focal length of the ARdevice according to an embodiment of the disclosure, and FIG. 12Billustrates a virtual image displayed on a real world object through theAR device.

Referring to FIG. 12A, the AR device 100 may project light of thevirtual object VO which provides information about the real world objectviewed by a user in the virtual image onto the waveguide 130 and adjustthe refractive power of the first focus adjustment region 112 of thefirst variable focus lens 110, thereby adjusting the vergence of thefirst variable focus lens 110. Because the vergence of the first focusadjustment region 112 is adjusted, the focal length of the virtualobject VO may be adjusted.

The AR device 100 may adjust the vergence of the first focus adjustmentregion 112 by adjusting the refractive power of the first focusadjustment region 112. The vergence is an index indicating a degree towhich light passing through the first focus adjustment region 112converges or diverges. Because the vergence is adjusted, the path of thelight incident on the lens 32 changes, and accordingly the position ofthe focal point on the retina 34 may change. Thus, the focal length maybe adjusted.

In the embodiment of the disclosure shown in FIG. 12A, the AR device 100may adjust the vergence of the first focus adjustment region 112,thereby the focal distance of the virtual object VO displayed throughthe waveguide 130. The AR device 100 may obtain the gaze point G inwhich a direction of view of the left eye and a direction of view of theright eye converge using a binocular disparity, the direction of view ofthe left eye and the direction of view of the right eye, and calculatethe vergence distance d_(con) that is a distance between both eyes andthe gaze point G using a triangulation method. In an embodiment of thedisclosure, the AR device 100 may change the focal length d_(f), whichis a physical distance between the waveguide 130 on which the virtualobject VO is displayed and both eyes, based on the vergence distanced_(con). In an embodiment of the disclosure, the processor 160 of the ARdevice 100 may adjust the vergence of the first focus adjustment region112 to shift the focal length converged to the virtual object VO throughboth eyes by Δd such that the focal length is the same as the vergencedistance d_(con). Here, the distance Δd is a distance at which the userdoes not feel dizzy when observing the virtual object VO while wearingthe AR device 100, and may be calculated by the processor 160 and storedin the storage 190 (FIG. 3).

In an embodiment of the disclosure, the first focus adjustment region112 may perform functionality similar to a function of a concave lensthat reflects the light path to form a negative vergence, and thusincreases the light path passing through the lens 32 and changes thefocus position formed on the retina 34. When the negative vergence isformed, the focal length of the virtual object VO may increase. In anembodiment of the disclosure, to compensate for a focus distortion inwhich the real world object or a physical space looks dim or blurry dueto the adjusted refractive power of the first focus adjustment region112, the AR device 100 may adjust the refractive power of the secondfocus adjustment region 122 such that the vergence of the second focusadjustment region 122 of the second variable focus lens 120 iscomplementarily formed with respect to the vergence of the first focusadjustment region 112. A method of complementarily adjusting therefractive power of the second focus adjustment region 122 with respectto the refractive power of the first focus adjustment region 112 isdescribed with reference to FIGS. 1 and 3, and thus redundantdescriptions thereof will be omitted.

Referring to FIG. 12B, the AR device 100 may display, on the real worldobject 10, the virtual object VO indicating information related to thereal world object 10 viewed through the first variable focus lens 110,the second variable focus lens 120, and the waveguide 130. The virtualobject VO refers to a part of the virtual image output to the user's eyethrough diffraction by the waveguide 130.

In an embodiment of the disclosure, the AR device 100 may obtain animage by capturing a physical environment or space viewed by the userthrough a camera 170 (see FIG. 12A) and provide the obtained image tothe processor 160 (see FIG. 12A). The processor 160 may recognize thereal world object 10 in the obtained image through image processing andgenerate the virtual object VO including the information related to thereal world object 10. In the embodiment of the disclosure shown in FIG.12B, the AR device 100 may obtain an image of a subway entrance bycapturing the subway entrance through the camera 170, and the processor160 may recognize the real world object 10 which describes subway lineand exit information at the subway entrance and generate the virtualobject VO including information related to the recognized real worldobject 10, i.e., information that ‘the subway service is stopped’.

In an embodiment of the disclosure, the AR device 100 may obtainposition information of the user wearing the AR device 100 using theposition sensor 180 (see FIG. 3) and generate the virtual object VOincluding information related to the position information. For example,the AR device 100 may generate the virtual object VO including a roadguidance UI, a navigation UI, etc.

The AR device 100 may project the generated virtual image onto thewaveguide 130 and adjust the refractive power of the first focusadjustment region 112, thereby forming the vergence of the first focusadjustment region 112 and accordingly changing the focal length of thevirtual object VO, which is a region of the virtual image formed on theretina 34.

FIG. 13A is a diagram for describing a method performed by the AR deviceof adjusting a focal length according to an embodiment of thedisclosure, and FIG. 13B is a diagram of a virtual image displayed on areal world object through the AR device.

Referring to FIG. 13A, an eye tracker may obtain an eye vector bytracking a direction of view of a user's eye and provide the eye vectorto the processor 160. The processor 160 may obtain the gaze point G onthe real world object 10 that the user views based on the eye vector.According to an embodiment of the disclosure, the processor 160 mayobtain a first eye vector indicating a direction of view of the left eyeand a second eye vector indicating a direction of view of the right eyeusing the eye tracker. The processor 160 may calculate a 3D positioncoordinate value of the gaze point G at which views of both eyesconverge based on a binocular disparity, the first eye vector, and thesecond eye vector. The processor 160 may calculate the vergence distanced_(con), which is a distance between both eyes and the gaze point G. Inan embodiment of the disclosure, the vergence distance d_(con) may bedetermined by a triangulation method.

The processor 160 may recognize the real world object 10 viewed by theuser based on the gaze point G and obtain depth value information aboutthe real world object 10. In an embodiment of the disclosure, the depthsensor 172 may scan a physical space or environment around the user andmeasure the depth value of the real world object 10 disposed in thephysical space or environment. The depth sensor 172 may measure thedepth value according to the 3D position coordinate value of the realworld object 10 in the physical space or environment and arrange themeasured depth value according to each 3D position coordinate value togenerate a depth map. The depth sensor 172 may measure the depth valueof the real world object 10 by using any one of, for example,stereo-type, ToF, and structured pattern techniques.

The depth sensor 172 may store the depth map in the storage 190 (seeFIG. 3). The processor 160 may load the depth map from the storage 190to obtain the depth information, which indicates a distance between theuser and the real world object 10 disposed on the gaze point G.

In an embodiment of the disclosure, the depth sensor 172 may obtain the3D position coordinate value of the gaze point G from the processor 160,measure the depth value corresponding to the obtained 3D positioncoordinate value, and provide the measured depth value to the processor160.

The AR device 100 may adjust the vergence of the first focus adjustmentregion 112 by adjusting the refractive power of the first focusadjustment region 112 of the first variable focus lens 110. Theprocessor 160 may change the focus position of the virtual object VO,which is one region of a virtual image output by the waveguide 130, byadjusting the vergence of the first focus adjustment region 112. In anembodiment of the disclosure, the processor 160 may adjust the vergenceof the first focus adjustment region 112, thereby changing the lightpath passing through each lens 32 of both eyes, and accordingly changethe focus position of the virtual object VO formed on the retina 34,thereby adjusting the focal length of the virtual object VO. In anembodiment of the disclosure, the processor 160 may obtain the depthvalue of the real world object 10 disposed on the gaze point G byloading the depth map previously stored in the storage 190 and adjustthe focal length d_(f) of the virtual object VO output through thewaveguide 130 to be the same as the obtained depth value of the realworld object 10.

In an embodiment of the disclosure, the AR device 100 may adjust therefractive power of the first focus adjustment region 112, therebychanging the focal length d_(f) of the virtual object VO output throughthe waveguide 130 to be the same as the vergence distance d_(con). Inthis case, the first focus adjustment region 112 may be adjusted to havea refractive power by which the focal length focused on the virtualobject VO output through the waveguide 130 may be shifted by the samedistance as the depth value of the real world object 10.

In an embodiment of the disclosure, the AR device 100 may adjust therefractive power of the first focus adjustment region 112 to place thefocus position converging on the virtual object VO within a targetposition T, and accordingly adjust the focus position of the virtualobject VO. The target position T represents a position within apredetermined range from the position of the gaze point G at which viewsof both eyes converge on the real world object 10. In this case, thefocal length d_(f) of the virtual object VO may not be exactly the sameas the depth value of the real world object 10. In an embodiment of thedisclosure, the focal length d_(f) may be larger or smaller than thedepth value of the real world object 10 by a predetermined distance.

In an embodiment of the disclosure, to compensate for a focus distortionin which the real world object or a physical space looks dim or blurrydue to the adjusted refractive power of the first focus adjustmentregion 112 by adjusting the refractive power of the first focusadjustment region 112, the AR device 100 may adjust the refractive powerof the second focus adjustment region 122 such that the vergence of thesecond focus adjustment region 122 of the second variable focus lens 120is complementarily formed with respect to the vergence of the firstfocus adjustment region 112. A method of complementarily adjusting therefractive power of the second focus adjustment region 122 with respectto the refractive power of the first focus adjustment region 112 isdescribed with reference to FIGS. 1 and 3, and thus redundantdescriptions thereof will be omitted.

Referring to FIG. 13B, the AR device 100 may capture a physical space orenvironment around the user wearing the AR device 100 using the depthsensor 172 (see FIG. 3) and obtain depth value information of each of aplurality of first through third real world objects 12, 14, and 16included in the physical space or environment.

The AR device 100 may use the depth value information of each of theplurality of first through third real world objects 12, 14, and 16 tochange the focal distance such that virtual objects VO1, VO2, and VO3are the same as depth values of the plurality of first through thirdreal world objects 12, 14, and 16 or are displayed within predeterminedranges of the depth values respectively.

In an embodiment of the disclosure, the AR device 100 may use the depthsensor 172 to measure the depth values according to 3D positioncoordinate values of the first through third real world objects 12, 14,and 16 in the physical space or environment viewed by the user andarrange the measured depth values according to the respective 3Dposition coordinate values to generate a depth map. The generated depthmap may be stored in the storage 190 (see FIG. 3). The processor 160(see FIG. 13A) may load the depth map from the storage 190 to obtaindepth information which is distances between the first through thirdreal world objects 12, 14, and 16 and the user. In an embodiment of thedisclosure, the processor 160 may obtain the depth value which is thedistance between the first real world object 12 and the user from thestorage 190 and adjust the refractive power of the first focusadjustment region 112 such that the focal length of the first virtualobject VO1 is the same as the depth value of the first real world object12. The processor 160 may adjust the refractive index of light passingthrough the first virtual object VO1 by adjusting the refractive powerof the first focus adjustment region 112, thereby changing the vergenceof the first variable focus lens 110 and accordingly adjust the focallength of the first virtual object VO1.

In the embodiment of the disclosure shown in FIG. 13B, positions wherethe first real world object 12, the second real world object 14, and thethird real world object 16 are disposed are different from each other,and thus, depth values with respect to the first real world object 12,the second real world object 14 and the third real world object 16 aredifferent. In an embodiment of the disclosure, the AR device 100 maytrack the direction of the user's view by using an eye tracker, therebyobtaining a gaze point at which user's both eyes converge among thefirst real world object 12, the second real world object 14 and thethird real world object 16 and adjusting the refractive power of thefirst focus adjustment region 112 using the depth value of the real wordobject disposed on the gaze point. The depth values of the first realworld object 12, the second real world object 14 and the third realworld object 16 may be obtained using the depth map previously stored inthe storage 190. The AR device 100 may adjust the refractive power ofthe second focus adjustment region 122 to form a complementary vergencewith respect to the vergence formed due to the adjusted refractive powerof the first focus adjustment region 112.

The AR device 100 may recognize the plurality of first through thirdreal world objects 12, 14, and 16, and generate the virtual objects VO1,VO2, and VO3 including information about the plurality of first throughthird real world objects 12, 14, and 16 respectively. For example, theAR device 100 may generate the virtual objects VO1, VO2, and VO3including at least one of detailed descriptions, price information,discount information, purchaser's website addresses, user ratings, oradvertisements regarding the plurality of first through third real worldobjects 12, 14, and 16. In the embodiment of the disclosure illustratedin FIG. 13B, the AR device 100 may recognize that the first real worldobject 12 is shoes, generate the first virtual object VO1 includingmembership discounts, coupon downloads, etc. related to the shoes, anddisplay the first virtual object VO1 through the waveguide 130. The ARdevice 100 may also generate the second virtual object VO2 includingprice information and user rating information about the second realworld object 14 and display the second virtual object VO2 through thewaveguide 130. The AR device 100 may recognize that the third real worldobject 16 is a shirt, generate the third virtual object VO3 includingprice information and about user rating information about the shirt, anddisplay the third virtual object VO3 through the waveguide 130.

FIG. 14 is a flowchart of a method performed by an AR device ofadjusting a refractive index of a focus adjust adjustment region of avariable focus lens based on a depth value of a real world objectaccording to an embodiment of the disclosure.

In operation S1410, the AR device may measure the depth value of thereal world object disposed on a gaze point using a depth camera. In anembodiment of the disclosure, a depth sensor may scan a physical spaceor environment around a user wearing the AR device and measure the depthvalue of the real world object disposed in the physical space orenvironment. The depth sensor may measure the depth value according tothe 3D position coordinate value of the real world object in thephysical space or environment and arrange the measured depth valueaccording to each 3D position coordinate value to generate a depth map.The depth sensor may measure the depth value of the real world object byusing any one of, for example, stereo-type, ToF, and structured patterntechnique. The depth sensor may store the generated depth map in thestorage 190 (see FIG. 3). The AR device may load the depth map from thestorage 190 to obtain the depth information, which is a distance betweenthe user and the real world object disposed on the gaze point.

In an embodiment of the disclosure, the depth sensor may obtain the 3Dposition coordinate value of the gaze point from the processor 160 (seeFIG. 3), and measure the depth value corresponding to the obtained 3Dposition coordinate value and provide the measured depth value to theprocessor 160.

In operation S1420, the AR device may adjust the refractive power of afirst focus adjustment region to adjust the focal length of a virtualobject based on the measured depth value. The AR device may adjust thefocal length of the virtual object corresponding to a region of avirtual image based on depth value information of the real world objectobtained by loading the depth value information from the storage 190.The AR device may adjust the refractive power of the first focusadjustment region of a first variable focus lens such that the depthvalue of the real world object and the focal length converged on thevirtual object are the same. In an embodiment of the disclosure, the ARdevice may adjust the refractive power of the first focus adjustmentregion of the first variable focus lens to adjust the focal length ofthe virtual object to be larger or smaller by a predetermined size withrespect to the depth value of the real world object.

The AR device may adjust a vergence formed in the first focus adjustmentregion by adjusting the refractive power of the first focus adjustmentregion. The vergence is an index indicating a degree of convergence ordivergence of light and may be adjusted according to the refractivepower of the first focus adjustment region. The AR device may change thelight path passing through a lens through the virtual object byadjusting the vergence of the first focus adjustment region andaccordingly change the focus position of the virtual object formed onthe retina, thereby adjusting the focal length of the virtual object.

In operation S1430, the AR device may complementarily adjust therefractive power of a second focus adjustment region with respect to therefractive power of the first focus adjustment region to offset thefocus distortion of the real world object by the adjusted refractivepower of the first focus adjustment region. To compensate for a focusdistortion in which the real world object looks dim or blurry thatoccurs by the vergence formed due to the adjusted refractive power ofthe first focus adjustment region of the first variable focus lens, theAR device may adjust the refractive power of the second focus adjustmentregion of a second variable focus lens to be the same as the adjustedrefractive power of the first focus adjustment region in a directionopposite to the direction of the refractive power of the first focusadjustment region. For example, when the first focus adjustment regionis adjusted to the refractive power of −1 diopter D, the second focusadjustment region may be adjusted to +1 diopter D.

FIG. 15A is a diagram illustrating a method performed by an AR device ofdetermining a size of a focus adjustment region of a variable focus lensaccording to an embodiment of the disclosure.

Referring to FIG. 15A, the first focus adjustment region 112 of thefirst variable focus lens 110, the second focus adjustment region 122 ofthe second variable focus lens 120, and the virtual object VO outputthrough the waveguide 130 may be aligned in a line according to an eyevector. In an embodiment of the disclosure, the AR device may determinethe size of the first focus adjustment region 112 based on the size ofthe virtual object VO output through the waveguide 130. In an embodimentof the disclosure, the AR device may determine the size of the firstfocus adjustment region 112 based on a distance between the user's eye30 and the first variable focus lens 110 and the size of the virtualobject VO.

When a real world object is viewed through the eye 30, a viewing anglemay gradually increase in a direction from the eye 30 toward the realworld object. The AR device may determine the size of the second focusadjustment region 122 based on the size of the virtual object VO and afirst distance Δd1 spaced apart between the first variable focus lens110 and the second variable focus lens 120.

FIG. 15B is a diagram illustrating a method performed by an AR device ofdetermining a size of a focus adjustment region of a variable focus lensaccording to an embodiment of the disclosure.

Referring to FIG. 15B, even when the viewing angle of the eye 30 is thesame, when a second distance Δd1 spaced apart between the first variablefocus lens 110 and the second variable focus lens 120 is larger than thefirst distance Δd1 (see FIG. 15A), the size of a second focus adjustmentregion 123 may be determined to be larger than the size of the secondfocus adjustment region 122 shown in FIG. 15A. In FIG. 15B, the size ofa first focus adjustment region 113 may be determined based on the sizeof the virtual object VO and a distance between the first variable focuslens 110 and the waveguide 130.

FIG. 16 illustrates an embodiment of the disclosure in which a variablefocus lens includes a plurality of first through fourth focus adjustmentregions.

Referring to FIG. 16, the first variable focus lens 110 may include theplurality of first and third focus adjustment regions 112 and 116, andthe second variable focus lens 120 may include the plurality of secondand fourth focus adjustment regions 122 and 126. In FIG. 16, the firstvariable focus lens 110 may include the two first and third focusadjustment regions 112 and 116, and the second variable focus lens 120may include the two second and fourth focus adjustment regions 122 and126, but this is for convenience of explanation. In the disclosure, thequantities of focus adjustment regions included in the first variablefocus lens 110 and the second variable focus lens 120 are not limited asillustrated in FIG. 16. In an embodiment of the disclosure, the firstvariable focus lens 110 may include three or more focus adjustmentregions, and the second variable focus lens 120 may also include threeor more focus adjustment regions.

The second focus adjustment region 122 of the second variable focus lens120 may be disposed at a position aligned with the first focusadjustment region 112 along the direction of a first eye vector p. In anembodiment of the disclosure, the processor 160 (see FIGS. 2 and 3) mayobtain a 2D position coordinate value of a region in which the first eyevector p arrives in the entire region of the second variable focus lens120 and determine a preset area region around the 2D position coordinatevalue as the second focus adjustment region 122.

The fourth focus adjustment region 126 of the second variable focus lens120 may be disposed at a position aligned with the third focusadjustment region 116 along the direction of a second eye vector p′. Inan embodiment of the disclosure, the processor 160 may obtain a 2Dposition coordinate value of a region in which the second eye vector p′arrives in the entire region of the second variable focus lens 120 anddetermine a preset area region around the 2D position coordinate valueas the fourth focus adjustment region 126.

The refractive power of the first focus adjustment region 112 and thethird focus adjustment region 116 may be adjusted to have differentrefractive indices. In an embodiment of the disclosure, the AR devicemay change the light path passing through the first focus adjustmentregion 112 by adjusting the first focus adjustment region 112 to a firstrefractive power such that the virtual object viewed through the firstfocus adjustment region 112 is displayed on a first focus distance.Through this, the vergence of the first focus adjustment region 112 maybe adjusted, and the focal length may be adjusted. In addition, the ARdevice may adjust the light path passing through the third focusadjustment region by adjusting the third focus adjustment region 116 toa second refractive index such that the virtual object viewed throughthe third focus adjustment region 116 is displayed on a second focallength. Through this, the vergence of the third focus adjustment region116 may be adjusted, and the focal length may be adjusted.

In an embodiment of the disclosure, the AR device may adjust therefractive power of each of the plurality of first and third focusadjustment regions 112 and 116 on the first variable focus lens 110based on a depth value of each of a plurality of real word objects 12and 14 disposed at positions corresponding to the plurality of first andthird focus adjustment regions 112 and 116 at the user's view. The depthvalue of each of the plurality of first and second real world objects 12and 14 may be obtained by loading a depth map previously stored in thestorage 190 (see FIG. 3). The depth map may be generated by scanning theplurality of first and second real world objects 12 and 14 by the depthsensor 172 (see FIG. 3) and storing a depth value corresponding to eachof 3D position coordinate values of the plurality of first and secondreal world objects 12 and 14.

For example, the AR device may adjust the refractive power of the firstfocus adjustment region 112 to obtain a first depth value depth1 of thefirst real world object 12 viewed in the direction of view of the firsteye vector p using the depth sensor and set the focal distance of thevirtual object to be the same as the first depth value depth1 about thefirst real world object 12. For example, the AR device may adjust therefractive power of the third focus adjustment region 116 to obtain asecond depth value depth2 of the second real world object 14 viewed inthe direction of view of the second eye vector p′ and set the focaldistance of the virtual object to be the same as the second depth valuedepth2 about the second real world object 14.

In an embodiment of the disclosure, the AR device may adjust therefractive power of the plurality of first and third focus adjustmentregions 112 and 116 by modulating a phase profile of a control voltageapplied through the plurality of first array excitation electrodes 640-1to 640-5 (see FIG. 6B) and the plurality of second array excitationelectrodes 650-1 to 650-5 (see FIG. 6B) disposed on the liquid crystallayer 610 (see FIGS. 6A and 6B) of the first variable focus lens 110.For example, to adjust the refractive power of each of the plurality offirst and third focus adjustment regions 112 and 116, when applying thecontrol voltage through a redundant excitation electrode among theplurality of first array excitation electrodes 640-1 to 640-5 and theplurality of second array excitation electrodes 650-1 to 650-5, the ARdevice may perform time division on the control voltage through theplurality of first driver terminals 660-1 to 660-5 (see FIG. 6B) and theplurality of second driver terminals 670-1 to 670-5 (see FIG. 6B) andcontrol the control voltage to be alternately applied to the excitationelectrode according to the time division.

The plurality of first driver terminals 660-1 to 660-5 (see FIG. 6B) andthe plurality of second driver terminals 670-1 to 670-5 (see FIG. 6B)may be controlled by the controller 680 (see FIG. 6B). The processor(see FIG. 6B) may provide the controller 680 with position informationabout the first focus adjustment region 112 and the second focusadjustment region 122, and the controller 680 may modulate the phaseprofile of the control voltage applied through the plurality of firstarray excitation electrodes 640-1 to 640-5 and the plurality of secondarray excitation electrodes 650-1 to 650-5 by controlling the pluralityof first driver terminals 660-1 to 660-5 and the plurality of seconddriver terminals 670-1 to 670-5 based on the position information aboutthe first focus adjustment region 112 and the second focus adjustmentregion 122 obtained from the processor 160.

In an embodiment of the disclosure, the AR device may adjust therefractive power of the second focus adjustment region 122 to compensatefor the adjusted refractive power of the first focus adjustment region112. To compensate for a focus distortion in which the first real worldobject 12 looks dim or blurry due to the adjusted refractive power ofthe first focus adjustment region 112, the AR device may complementarilyadjust the refractive power of the second focus adjustment region 122 ofthe second variable focus lens 120 with respect to the refractive powerof the first focus adjustment region 112. For example, the AR device mayadjust the refractive power of the second focus adjustment region 122 tohave the same magnitude as the adjusted refractive power of the firstfocus adjustment region 112 in a direction opposite to the direction ofthe refractive power of the first focus adjustment region 112. Forexample, when the first focus adjustment region 112 is adjusted to therefractive power of −1 diopter D, the second focus adjustment region 122may be adjusted to +1 diopter D.

Similarly, to compensate for the adjusted refractive power of the thirdfocus adjustment region 116, the AR device may complementarily adjustthe refractive power of the fourth focus adjustment region 126 withrespect to the refractive power of the third focus adjustment region116.

When a plurality of focus adjustment regions are set on the firstvariable focus lens 110 and the second variable focus lens 120, the ARdevice may receive virtual game content or 3D moving image content froma server or the like and adjust a focus on an object including acharacter displayed on the received content. In an embodiment of thedisclosure, when the AR device receives game content or 3D moving imagecontent from a server or another device and executes the receivedcontent or executes game content or 3D moving image content previouslystored in the AR device, the first focus adjustment region 112 and thethird focus adjustment region 116 may be previously adjusted to apredetermined refractive power. In this case, information about therefractive power of the first focus adjustment region 112 and therefractive power of the third focus adjustment region 116 may be storedin the storage 190 (see FIG. 3). Similarly, information about therefractive power of the second focus adjustment region 122 and thefourth focus adjustment region 126 may be previously stored in thestorage 190.

In an embodiment of the disclosure, the processor 160 (see FIG. 3) mayobtain an eye vector indicating a direction of the user's view using theeye tracker 150 (see FIG. 3), and based on eye vectors of the left andright eyes, obtain a gaze point at which views of both eyes converge.The processor 160 may determine one of the plurality of first and thirdfocus adjustment regions 112 and 116 included in the first variablefocus lens 110 based on the detected gaze point. The processor 160 mayobtain information about the refractive power of the determined focusadjustment region from the storage 190 and adjust the refractive powerof the focus adjustment region using the obtained information about therefractive power.

For example, when the eye vector obtained through the eye tracker 150 isthe first eye vector p, the processor 160 may determine the refractivepower of the first focus adjustment region 112 as a focus adjustmentregion to be adjusted and adjust the refractive power of the first focusadjustment region 112 using the information about the refractive powerof the first focus adjustment region 112 previously stored in thestorage 190. To compensate for a focus distortion due to the adjustedrefractive power of the first focus adjustment region 112, the processor160 may complementarily adjust the refractive power of the second focusadjustment region 122 with respect to the refractive power of the firstfocus adjustment region 112.

FIG. 17 is a block diagram illustrating an AR device according to anembodiment of the disclosure.

Referring to FIG. 17, the AR device 200 may include the first variablefocus lens 110, the second variable focus lens 120, the waveguide 130,the display module 140, the eye tracker 150, the processor 160, thememory 162, and a communication module 270. The AR device 200illustrated in FIG. 17 may include the same elements as the AR device100 illustrated in FIG. 3. For example, the first variable focus lens110, the second variable focus lens 120, the waveguide 130, the displaymodule 140, and the eye tracker 150 are respectively the same as thefirst variable focus lens 110, the second variable focus lens 120, thewaveguide 130, the display module 140, and the eye tracker 150 shown inFIG. 3. Therefore, redundant descriptions will be omitted.

The processor 160 may control the overall functions and/or operationsperformed by the AR device 200 by executing computer program codeincluding instructions stored in the memory 162. The processor 160 maycontrol operations or functions of the first variable focus lens 110,the second variable focus lens 120, the waveguide 130, the displaymodule 140, the eye tracker 150, and the communication module 270.

The communication module 270 may perform data communication between theAR device 200 and a mobile phone 1000. The AR device 200 may bewirelessly connected to the mobile phone 1000 through the communicationmodule 270.

The communication module 270 may perform data communication between theAR device 200 and the mobile phone 1000 by using at least one of datacommunication methods including wireless LAN, Wi-Fi, Bluetooth, Zigbee,Wi-Fi Direct (WFD), infrared communication (IrDA), Bluetooth Low Energy(BLE) Near Field Communication (NFC), Wireless Broadband Internet(Wibro), World Interoperability for Microwave Access (WiMAX), SharedWireless Access Protocol (SWAP), Wireless Gigabit Alliance (Wigig) or RFcommunication.

The mobile phone 1000 may be operated by a user wearing the AR device200. The mobile phone 1000 may obtain a real world object image bycapturing a physical space or environment around the user through acamera. The mobile phone 1000 may obtain information about the positionof the user by using a location sensor such as a GPS sensor. The mobilephone 1000 may generate a virtual image by using information about areal world object. In an embodiment of the disclosure, the mobile phone1000 may include a depth sensor and obtain depth value information ofthe real world object.

The processor 160 may control the communication module 270 and receiveat least one of location information, real world object information, ordepth value information of the real world object from the mobile phone1000 through the communication module 270. The processor 160 may controlthe display module 140 to project light of the virtual image onto thewaveguide 130 based on the received location information and the realworld object information. The virtual object which is a region of thevirtual image may be diffracted through the diffraction grating of thewaveguide 130 and output to the user's eye.

In an embodiment of the disclosure, the AR device 200 may receive fromthe mobile phone 1000 a virtual object generated by the mobile phone1000 using the communication module 270. The processor 160 may controlthe display module 140 to project the received virtual object toward thewaveguide 130.

In an embodiment of the disclosure, the AR device 200 may receive thedepth value information of the real world object from the mobile phone1000 using the communication module 270. The processor 160 may adjustthe refractive power of a focus adjustment region of the first variablefocus lens 110 to change the focal length of the virtual object based onthe depth value received from the mobile phone 1000. In an embodiment ofthe disclosure, the processor 160 may adjust the focal length of thevirtual object based on the depth value of the real world object viewedby the user by adjusting the refractive power of the focus adjustmentregion of the first variable focus lens 110.

The AR devices 100 and 200 according to the disclosure may be realizedas hardware elements, software elements, and/or the combination ofhardware elements and software elements. For example, the AR devices 100and 200 according to the embodiments of the disclosure may be realizedby using a processor, an arithmetic logic unit (ALU), applicationspecific integrated circuits (ASICs), digital signal processors (DSPs),digital signal processing devices (DSPDs), programmable logic devices(PLDs), microcomputers, microprocessors, or one or more general-purposecomputers or special-purpose computers, such as a device capable ofexecuting and responding to instructions.

The software may include a computer program, a code, an instruction, ora combination of one or more thereof, and may configure a processingdevice to operate as required or separately or collectively command theprocessing device.

The software may be implemented in a computer program that includesinstructions stored on a computer-readable storage medium. Thecomputer-readable storage media may include, for example, magneticstorage media (for example, ROM, RAM, floppy disks, hard disks, etc.)and optical reading media (for example, CD-ROM, DVD, etc.). Thecomputer-readable recording media may be distributed in computer systemsconnected in a network and may store and execute computer-readable codesin a distributed fashion. The media may be computer-readable, may bestored in a memory, and executed by a processor.

The computer may be a device configured to call instructions stored inthe storage media, and in response to the called instructions, toperform an operation according to the embodiments of the disclosure, andmay include the AR devices 100 and 200 according to the embodiments ofthe disclosure.

The computer-readable storage medium may be provided in the form of anon-transitory storage medium. Here, ‘non-transitory’ means that thestorage medium does not include a signal and is tangible, but does notdistinguish whether data is stored semi-permanently or temporarily onthe storage medium.

Further, the AR devices 100 and 200 or the operating method of the sameaccording to the embodiments of the disclosure may be provided in acomputer program product. The computer program product is a productpurchasable between a seller and a purchaser.

The computer program product may include a software program and acomputer-readable storage medium in which the software program isstored. For example, the computer program product may include a softwareprogram-type product (for example, a downloadable application)electronically distributed by a manufacturer of the AR devices 100 and200 or electronic markets (for example, Google Play™ store, App Store,etc.). For electronic distribution, at least a portion of the softwareprogram may be stored in storage media or temporarily generated. In thiscase, the storage media may be a server of the manufacturer, a server ofthe electronic market, or a storage medium of a broadcasting servertemporarily storing the software program.

The computer program product may include a storage medium of a server ora storage medium of a terminal in a system including the server and theterminal (for example, an ultrasonic diagnosis apparatus).Alternatively, when there is a third device (for example, a smartphone)connected with the server or the terminal for communication, thecomputer program product may include a storage medium of the thirddevice. Alternatively, the computer program product may include asoftware program transmitted to the terminal or the third device fromthe server or to the terminal from the third device.

In this case, one of the server, the terminal, and the third device mayexecute the method according to the embodiments of the disclosure byexecuting the computer program product. Alternatively, at least two ofthe server, the terminal, and the third device may execute the methodaccording to the embodiments of the disclosure in a distributed fashionby executing the computer program product.

For example, the server (for example, a cloud server or an AI server)may execute the computer program product stored in the server andcontrol the terminal connected with the server for communication toperform the method according to the embodiments of the disclosure.

As another example, the third device may execute the computer programproduct and control the terminal connected to the third device forcommunication to perform the method according to the embodiments of thedisclosure.

When the third device executes the computer program product, the thirddevice may download a computer program product from the server andexecute the downloaded computer program product. Alternatively, thethird device may execute the computer program product provided in afree-loaded state and perform the method according to the embodiments ofthe disclosure.

In addition, although the embodiments of the disclosure have beenillustrated and described above, the disclosure is not limited to theabove-described specific embodiments of the disclosure. Various modifiedembodiments of the disclosure may be made by one of ordinary skill inthe art without departing from the scope of the disclosure as claimed inthe claims, and these modifications should not be individuallyunderstood from the technical spirit or the prospect of the disclosure.

Although the embodiments of the disclosure have been described by thelimited embodiments of the disclosure and the drawings as describedabove, various modifications and variations are possible by one ofordinary skill in the art from the above description. For example, thedescribed techniques may be performed in a different order than thedescribed method, and/or elements of the described electronic device,structure, circuit, etc. may be combined or integrated in a differentform than the described method, or may be replaced or substituted byother elements or equivalents to achieve appropriate results.

What is claimed is:
 1. An augmented reality (AR) device comprising: aplurality of first variable focus lenses and a plurality of secondvariable focus lenses, wherein each of the plurality of first variablefocus lenses and each of the plurality of second variable focus lensesis an electrically tunable liquid crystal lens; a plurality ofwaveguides, wherein each of the plurality of waveguides is disposedbetween the plurality of first variable focus lenses and the pluralityof second variable focus lenses; a plurality of eye trackers configuredto obtain a left eye vector by tracking a direction of a left eye of auser of the AR device and a right eye vector by tracking a direction ofa right eye of the user of the AR device; a display module configured toproject light of a virtual image toward the plurality of waveguides; andone or more processors configured to: determine a first focus adjustmentregion of a first left eye variable focus lens among the plurality offirst variable focus lenses based on the left eye vector determine athird focus adjustment region of a first right eye variable focus lensamong the plurality of first variable focus lenses based on the righteye vector, obtain a gaze point based on the left eye vector and theright eye vector, adjust a refractive power of the first focusadjustment region and a refractive power of the third focus adjustmentregion based on the gaze point, and based on the adjusted refractivepower of the first focus adjustment region and the adjusted refractivepower of the third focus adjustment region, adjust a refractive power ofa second focus adjustment region of a second left eye variable focuslens among the plurality of second variable focus lenses and arefractive power of a fourth focus adjustment region of a second righteye variable focus lens among the plurality of second variable focuslenses.
 2. The AR device of claim 1, further comprising a memory storinga program comprising one or more instructions, wherein the one or moreprocessors are further configured to perform operations by executing theone or more instructions of the program stored in the memory.
 3. The ARdevice of claim 1, wherein when the user wears the AR device, the firstleft eye variable focus lens is disposed at a position spaced apart fromthe left eye of the user by a first distance, and the second left eyevariable focus lens is disposed at a position spaced apart from the lefteye of the user by a second distance, wherein the second distance isgreater than the first distance.
 4. The AR device of claim 1, whereinthe one or more processors are further configured to control to apply acontrol voltage that generates a phase modulation profile relating to aposition corresponding to the first focus adjustment region to the firstleft eye variable focus lens, and, based on the applied control voltage,adjust the refractive power of the first focus adjustment region bychanging an angle at which liquid crystal molecules arranged at aposition of the first focus adjustment region are arranged among liquidcrystal molecules of the first left eye variable focus lens.
 5. The ARdevice of claim 1, further comprising a depth sensor configured tomeasure a depth value of a real world object disposed at the gaze point,wherein the one or more processors are further configured to obtain themeasured depth value of the real world object from the depth sensor andadjust the refractive power of the first focus adjustment region basedon the measured depth value so as to adjust a focal length of a virtualobject which is a partial region of the virtual image.
 6. The AR deviceof claim 1, wherein the one or more processors are further configured toadjust the refractive power of the second focus adjustment region to bethe same as the refractive power of the first focus adjustment region ina direction opposite to a direction of the refractive power of the firstfocus adjustment region.
 7. The AR device of claim 1, wherein the one ormore processors are further configured to determine the second focusadjustment region based on the left eye vector such that the first focusadjustment region and the second focus adjustment region are aligned ina direction of the left eye vector.
 8. The AR device of claim 1, whereina size of the first focus adjustment region is determined based on asize of a virtual object that is a partial region of the virtual imageoutput through a left eye waveguide.
 9. The AR device of claim 1,wherein a size of the second focus adjustment region is determined basedon a size of a virtual object that is a partial region of the virtualimage output through a left eye waveguide and a spaced distance betweenthe first left eye variable focus lens and the second left eye variablefocus lens.
 10. The AR device of claim 1, wherein a plurality of a firstfocus adjustment regions are provided on the first left eye variablefocus lens, and wherein the one or more processors are furtherconfigured to adjust refractive powers focus adjustment regions suchthat different vergences are formed according to the focus adjustmentregions, the focus adjustment regions comprising at least one of thefirst focus adjustment region, the second focus adjustment region, thethird focus adjustment region, and the fourth focus adjustment region.11. An augmented reality (AR) device comprising: a plurality of firstvariable focus lenses and a plurality of second variable focus lenses,wherein each of the plurality of first variable focus lenses and each ofthe plurality of second variable focus lenses is an electrically tunableliquid crystal lens; a plurality of waveguides, wherein each of theplurality of waveguides is disposed between the plurality of firstvariable focus lenses and the plurality of second variable focus lenses;a plurality of eye trackers configured to obtain a left eye vector bytracking a direction of a left eye of an user of the AR device and aright eye vector by tracking a direction of a right eye of the user ofthe AR device; a display module configured to project light of a virtualimage toward the plurality of waveguides; and one or more processorsconfigured to: determine a first focus adjustment region of a first lefteye variable focus lens among the plurality of first variable focuslenses based on the left eye vector obtained from the plurality of eyetrackers and determine a third focus adjustment region of a first righteye variable focus lens among the plurality of first variable focuslenses based on the right eye vector, obtain a gaze point based on theleft eye vector and the right eye vector, adjust a refractive power ofthe first focus adjustment region and a refractive power of the thirdfocus adjustment region based on the gaze point, to change a focallength of a real world object, adjust a refractive power of a secondfocus adjustment region of a second left eye variable focus lens amongthe plurality of second variable focus lenses and a refractive power ofa fourth focus adjustment region of a second right eye variable focuslens among the plurality of second variable focus lenses, andindependently adjust the refractive power of the second focus adjustmentregion and the refractive power of the fourth focus adjustment regionregardless of the refractive power of the first focus adjustment regionand the refractive power of the third focus adjustment region.
 12. Anoperating method of an augmented reality (AR) device, the operatingmethod comprising: obtaining a left eye vector by tracking a directionof a left eye using a first eye tracker and obtaining a right eye vectorby tracking a direction of a right eye using a second eye tracker;determining a first focus adjustment region of a first left eye variablefocus lens based on the obtained left eye vector and determining a thirdfocus adjustment region of a first right eye variable focus lens basedon the obtained right eye vector; obtaining a gaze point based on theleft eye vector and the right eye vector, adjusting a refractive powerof the first focus adjustment region and a refractive power of the thirdfocus adjustment region based on the gaze point; based on the adjustedrefractive power of the first focus adjustment region and the adjustedrefractive power of the third focus adjustment region, adjusting arefractive power of a second focus adjustment region of a second lefteye variable focus lens and a refractive power of a fourth focusadjustment region of a second right eye variable focus lens; andprojecting light of a virtual image toward a waveguide using a displaymodule.
 13. The operating method of claim 12, wherein the adjusting therefractive power of the first focus adjustment region and the refractivepower of the third focus adjustment region comprises: applying a controlvoltage that generates a phase modulation profile relating to a positioncorresponding to the first focus adjustment region to the first left eyevariable focus lens; and based on the applied control voltage, adjustingthe refractive power of the first focus adjustment region by changing anangle at which liquid crystal molecules arranged at a position of thefirst focus adjustment region are arranged among liquid crystalmolecules of the first left eye variable focus lens.
 14. The operatingmethod of claim 12, wherein the adjusting of the refractive power of thefirst focus adjustment region and the refractive power of the thirdfocus adjustment region comprises: measuring a depth value of a realworld object disposed on the gaze point using a depth sensor; andadjusting the refractive power of the first focus adjustment regionbased on the measured depth value so as to adjust a focal length of avirtual object which is a partial region of the virtual image.
 15. Theoperating method of claim 12, wherein the adjusting of the refractivepower of the second focus adjustment region and the refractive power ofthe fourth focus adjustment region comprises: adjusting the refractivepower of the second focus adjustment region such that the second focusadjustment region forms a vergence in a direction opposite to adirection of an adjusted vergence of the first focus adjustment regionand adjusting the refractive power of the fourth focus adjustment regionsuch that the fourth focus adjustment region forms a vergence in adirection opposite to a direction of an adjusted vergence of the thirdfocus adjustment region.
 16. The operating method of claim 15, whereinthe adjusting of the refractive power of the second focus adjustmentregion and the refractive power of the fourth focus adjustment regioncomprises: adjusting the refractive power of the second focus adjustmentregion to be the same as adjusted refractive power of the first focusadjustment region in a direction opposite to a direction of the adjustedrefractive power of the first focus adjustment region.
 17. The operatingmethod of claim 12, further comprising determining the second focusadjustment region based on the left eye vector such that the first focusadjustment region and the second focus adjustment region are aligned ina direction of the left eye vector.
 18. The operating method of claim12, wherein a size of the first focus adjustment region is determinedbased on a size of a virtual object that is a partial region of thevirtual image output through the waveguide.
 19. The operating method ofclaim 12, wherein a size of the second focus adjustment region isdetermined based on a size of a virtual object that is a partial regionof the virtual image output through the waveguide and a spaced distancebetween the first left eye variable focus lens and the second left eyevariable focus lens.
 20. A non-transitory computer readable recordingmedium storing a program for executing the operating method of claim 12.